1.1. openMosix HOWTO

In the beginning there was Mosix, then came openMosix, in my opinion a more interesting project. Not only from a technical point of view but also due to the more correct license. I made the decision to focus this HOWTO on openMosix rather than on Mosix, mainly based on the fact that openMosix has a bigger userbase. (Moshe Bar states that about 97% of the old Mosix community has switched over to openMosix.) (20020705) Given the above, lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest release of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 My intention is to focus on the openMosix HOWTO, however not neglecting the Mosix users. More info on http://howto.ipng.be/Mosix-HOWTO/

1.2. Introduction

This document gives a brief description of openMosix, a software package that turns a network of GNU/Linux computers into a computer cluster. Along the way, some background to parallel processing is given, as well as a brief introduction to programs that make special use of openMosix's capabilities. The HOWTO expands on the documentation as it provides more background information and discusses the quirks of various distributions.

Since the creation of this HOWTO some people of the Mosix team created openMosix (more info later), initially both openMosix and Mosix were discussed in this HOWTO. Although lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest relase of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 and can be found on http://howto.ipng.be/Mosix-HOWTO/Mosix-HOWTO/

Kris Buytaert got involved in this piece of work when Scot Stevenson was looking for somebody to take over the Job: this was during February 2002. While initially we discussed both Mosix and openMosix, this version of the HOWTO now mainly focuses on openMosix. Please note that the document often still mentions Mosix where it should read openMosix.

You will notice that some of the headings are not as serious as they should be. Scot had planned to write the HOWTO in a slightly lighter style, as the world (and even the part of the world with a burping penguin as a mascot) is full of technical literature that is deadly. Therefore some parts still have these comments.

1.3. Disclaimer

Use the information in this document at your own risk. I disavow potential liability for the contents of this document. Use of these concepts, examples, and/or other content of this document is entirely at your own risk.

All copyrights are owned by their respective owners, unless specified otherwise. Use of a term in this document should not be regarded affecting the validity of any trademark or service mark. openMosix is Copyright (c) by Moshe Bar. Mosix is Copyright (c) by Amnon Barak. Linux is a Registered Trademark of Linus Torvalds. openMosix is licensed under version 2 of the GNU General Public License as published by the Free Software Foundation.

Naming of particular products or brands should not be seen as endorsements.

You are strongly recommended to take a backup of your system before major installation and backups at regular intervals.

1.4. Distribution policy

Copyright (c) 2002 by Kris Buytaert and Scot W. Stevenson. This document may be distributed under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the appendix entitled "GNU Free Documentation License".

1.5. New versions of this document

Official New versions of this document can be found on the web pages of the Linux Documentation Project Drafts and Beta versions will be available on howto.ipng.be in the appropriate sub folder. Changes to this document will usually be discussed on the openMosix Mailing Lists. See the openMosix for details.

1.6. Feedback

Currently this HOWTO is being maintained by Kris Buytaert. Please do send remarks and updates for the howto to him.

If you have a technical question about openMosix/Mosix itself, please post them on the more appropriate mailing list.

2.1. A very, very brief introduction to clustering

Most of the time, your computer is bored. Start a program like xload or top that monitors your system use, and you will probably find that your processor load is not even hitting the 1.0 mark. If you have two or more computers, chances are that at any given time, at least one of them is doing nothing. Unfortunately, when you really do need CPU power - during a C++ compile, or encoding Ogg Vorbis music files - you need a lot of it at once. The idea behind clustering is to spread these loads among all available computers, using the resources that are free on other machines.

The basic unit of a cluster is a single computer, also called a "node". Clusters can grow in size - they "scale" - by adding more machines. A cluster as a whole will be more powerful the faster the individual computers and the faster their connection speeds are. In addition, the operating system of the cluster must make the best use of the available hardware in response to changing conditions. This becomes more of a challenge if the cluster is composed of different hardware types (a "heterogeneous" cluster), if the configuration of the cluster changes unpredictably (machines joining and leaving the cluster), and the loads cannot be predicted ahead of time.

2.2. The story so far

2.3. openMosix in action: An example

openMosix clusters can take various forms. To demonstrate this, let's assume you are a student and share a dorm room with a rich computer science guy, with whom you have linked computers to form an openMosix cluster. Let's also assume you are currently converting music files from your CDs to Ogg Vorbis for your private use, which is legal in your country. Your roommate is working on a project in C++ that he says will bring World Peace. However, at just this moment he is in the bathroom doing unspeakable things, and his computer is idle.

So when you start a program like bladeenc to convert Bach's .... from .wav to .ogg format, the openMosix routines on your machine compare the load on both nodes and decide that things will go faster if that process is sent from your Pentium-233 to his Athlon XP. This happens automatically: you just type or click your commands as you would if you were on a standalone machine. All you notice is that when you start two more coding runs, things go a lot faster, and the response time doesn't go down.

Now while you're still typing ...., your roommate comes back, mumbling something about red chili peppers in cafeteria food. He resumes his tests, using a program called 'pmake', a version of 'make' optimized for parallel execution. Whatever he's doing, it uses up so much CPU time that openMosix even starts to send subprocesses to your machine to balance the load.

This setup is called *single-pool*: all computers are used as a single cluster. The advantage/disadvantage of this is that your computer is part of the pool: your stuff will run on other computers, but their stuff will run on yours too.

2.4. Components

2.5. openMosix Test Drive

In support of openMosix, Major Chai Mee Joon is giving OM users a free trial account to his online openMosix cluster service, which users can use to test and experiment openMosix with.

The availability of this online openMosix cluster service will help both new users overcome the initial openMosix configuration issues, and also provides higher computing power to openMosix users who are developing or porting their applications.

To get your userid and password to the cluster: http://www.mosixcluster.com/trial.php

2.6. Pros of openMosix

2.7. Cons of openMosix

3.1. Hardware requirements

Installing a basic cluster requires at least 2 network connected machines, either using a cross-cable between the two network cards or using a switch or hub (a switch is much better than a hub though and only costs a few bucks more). Of course the faster your network-cards the easier you will get better performance for your cluster.

These days Fast Ethernet (100 Mbps) is standard; putting multiple ports in a machine isn't that difficult, but make sure to connect them through other physical networks in order to gain the speed you want. Gigabit Ethernet is getting cheaper every day now but I suggest that you don't rush to the shop spending your money before you have actually tested your setup with multiple 100Mbit cards and noticed that you really do need the extra network capacity. Next to putting a Gigabit card you might also want to try bonding different 100Mbit cards together. An even cheaper alternative can be found in Firewire, as discussed in this paper

3.2. Hardware Setup Guidelines

Setting up a big cluster requires some thinking to be done: where are you going to put the machines? Not under a table somewhere or in the middle of your office I hope! It's ok if you just want to do some small tests, but if you are planning to deploy a N node cluster you will have to make sure that the environment that will hold these machines is capable of doing so.

I'm talking about preparing one or more 19" racks to host the machines, configure the appropriate network topology, either straight, single connected or even a 1 to 1 cross connected network between all your nodes. You will also need to make sure that there is enough power to support such a range of machines, that your air-conditioning system supports the load and that in case of power-failure your UPS can cleanly shut down all the required systems. You might want to invest in a KVM (Keyboard, Video, Mouse) Switch in order to facility access to the machines' consoles.

But even if you don't have the number of nodes that justifies such an investment, make sure that you can always easily access the different nodes, you never know when you have to replace a CPU fan or an hard-disk of a machine in trouble. If that means that you have to unload a stack of machines to reach the bottom one, hence shutting down your cluster, you are in trouble.

3.3. Software requirements

The systems we plan to use will need a basic Linux installation of your choice: Red Hat, Suse, Debian, Gentoo or any another distribution: it doesn't really matter which one. What does matter is that the kernel is at least on 2.4 level, and that your network-cards are configured correctly; next to that you'll need a healthy space of swap.

3.4. Planning your Cluster

When it comes to configuring openMosix Clusters with a pool of servers and a set of (personal) workstations, you have different options that will have their advantages and disadvantages.

• In a Single-pool configuration all the servers and workstations are used as a single cluster: each machine is a part of the cluster and can migrate processes to each other existing node. This of course makes your workstation a part of the pool.

• In an environment that is called a Server-pool, servers are a part of the cluster while workstations aren't part of it, they don't even haven openMosix kernel. If you want to run applications on the cluster you will need to specifically log on to these servers. However your workstation will also stay clean and no remote processes will migrate to it.

• A third alternative is called an Adaptive-pool configuration: here servers are shared while workstations join or leave the cluster. Imagine your workstation being used during daytime by yourself but, as soon as you log out in the evening, a script tells the workstation to join the cluster and start crunching numbers. This way your machine is being used while you don't need it. If you need the resources of the machine again just run the openmosix stop script and your processes will stay away from the cluster and vice-versa.

  Practically this means that you will change the role of your machine by using mosctl.

3.5. Classrooms

Although it might seem a good idea to convert your classroom into an openMosix cluster at night, you'll have to consider training your end users not to pull the power switch of those machines when they want to use them again. More recent machines support automatic shutdowns when hitting the power button, but with older machines you might loose some data now and then when this actually happens.

4.1. Installing openMosix

This chapter deals with installing openMosix on different distributions. It won't be an exhaustive list of all the possible combinations. However throughout the chapter you should find enough information on installing openMosix in your environment.

Techniques for installing multiple machines with openMosix will be discussed in one of the next chapters.

4.2. Getting openMosix

You can download the latest versions of openMosix from http://sourceforge.net/project/showfiles.php?group_id=46729. You can either choose the binary (even in rpm) compiled for UP or SMP or download the source code. You will need both the kernel patch or binaries and the userland tools. Alternatively you can get the CVS version:

cvs -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix login cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co linux-openmosix cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co userspace-tools

At the password prompt, just type enter since you're doing an anonymous login. Please take care that CVS trees DO BREAK now and then and that it might not be the easiest way to install openMosix ;-)

4.3. openMosix General Instructions

4.4. Red Hat and openMosix

If you are running a RedHat 7.2, 7.3 or 8.0 version, this is probably the easiest *Mosix install you have ever done. Choose the appropriate openMosix RPMs from sourceforge. They have precompiled kernels (as I write this 2.4.20) that work seamlessly: I have tested them on several machines including Laptops with PCMCIA cards and Servers with SCSI disks. If you are a grub user, the kernel rpm even modifies your grub.conf. So all you have to do is install 2 RPMs:

rpm -Uvh openmosix-kernel-2.4.20-openmosix2.i686.rpm openmosix-tools-0.2.4-1.i386.rpm

and edit your /etc/openmosix.map. Since this seems to be a problem for lots of people, let's go with another example. Say you have 3 machines: 192.168.10.220, 192.168.10.78 and 192.168.10.84. Your openmosix.map will look like this.

[root@oscar0 root]# more /etc/openmosix.map # openMosix CONFIGURATION # =================== # # Each line should contain 3 fields, mapping IP addresses to openMosix node-numbers: # 1) first openMosix node-number in range. # 2) IP address of the above node (or node-name from /etc/hosts). # 3) number of nodes in this range. # # Example: 10 machines with IP 192.168.1.50 - 192.168.1.59 # 1 192.168.1.50 10 # # openMosix-# IP number-of-nodes # ============================ 1 192.168.10.220 1 2 192.168.10.78 1 3 192.168.10.84 1

Now by rebooting the different machines with the newly installed kernel you will get one step closer to having a working cluster.

Most RedHat installations have one extra thing to fix. You often get the following error:

[root@inspon root]# /etc/init.d/openmosix start Initializing openMosix... setpe: the supplied table is well-formatted, but my IP address (127.0.0.1) is not there!

This means that your hostname is not listed in /etc/hosts with the same ip as in your openmosix.map. You might have a machine called omosix1.localhost.org in your hostfile listed as

127.0.0.1 omosix1.localhost.org localhost

If you modify your /etc/hosts to look like below, openMosix will have less troubles starting up.

192.168.10.78 omosix1.localhost.org 127.0.0.1 localhost

[root@inspon root]# /etc/init.d/openmosix start Initializing openMosix... [root@inspon root]# /etc/init.d/openmosix status This is openMosix node #2 Network protocol: 2 (AF_INET) openMosix range 1-1 begins at 192.168.10.220 openMosix range 2-2 begins at inspon.localhost.be openMosix range 3-3 begins at 192.168.10.84 Total configured: 3

If you would like to use more bleeding edge patches, you can always opt for the src rpm and run rpmbuild --rebuild on it. This will install the source for you and create an initial config file. From there you can go further applying patches to openMosix

As new RedHat versions come out, they might be supported out of the box so, feel free to drop the author a note and help him keeping this information updated.

4.5. Suse and openMosix

Although the RPMs are being built on a RedHat based environment, you can use most of them on other RPM based systems.

Suse however has /sbin/mk_initrd as a link to /sbin/mkinitrd, which makes rpms before release 20-2 fail. Newer version should have a fix for this.

4.6. Debian and openMosix

Installing openMosix ``the Debian way'' can be easily done as described below.

The first step consists in downloading the packages from the net. I had to use a 2.4.19 kernel since the openMosix patches package is not yet available for 2.4.20 at the moment I write this. Since we are using a Debian setup we needed: http://packages.debian.org/unstable/net/openmosix.html, http://packages.debian.org/unstable/net/kernel-patch-openmosix.html, http://packages.debian.org/unstable/misc/kernel-package.html, http://packages.debian.org/unstable/devel/kernel-source-2.4.19.html. You can also apt-get install them ;).

The next part is making the kernel openMosix capable.

Basically, the procedure to follow is:

cd /usr/src apt-get install kernel-source-2.4.19 kernel-package \ openmosix kernel-patch-openmosix tar vxjf kernel-source-2.4.19.tar.bz2 ln -s /usr/src/kernel-source-2.4.19 /usr/src/linux cd /usr/src/linux ../kernel-patches/i386/apply/openmosix make menuconfig make-kpkg kernel_image modules_image cd .. dpkg -i kernel-image-*-openmosix-*.deb

You now will need to edit your /etc/openmosix.map. Please follow the instructions given in the ``Syntax of /etc/openmosix.map'' part of this HOWTO.

After rebooting with this kernel and a configured /etc/openmosix.map, you should then have a cluster of openMosix machines that talk to each-other and that do migration of processes.

You can test that by running the following small script:

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}'

a couple of times, and monitor its behaviour with "mosmon" where you will see that it spreads the load between the different nodes.

We also setup openMosixView on the Debian machine:

apt-get install openmosixview

In order to be able to actually use openMosixView you will need to run it from a user who can log in to the different nodes as root. We suggest you set this up using ssh. Please note that there is a difference between the ssh and ssh2 implementations. If you do have an identity.pub file, ssh will check authorized_keys, while if you do have an id_dsa.pub you will need authorized_keys2!

openMosixView gives you a nice interface that shows the load of different machines and gives you the possibility to migrate processes manually.

A detailed discussion of openMosixView can be found elsewhere in this document.

4.7. openMosix and Gentoo

First Install Gentoo Linux

Then, install openMosix: type "emerge sys-apps/openmosix-user", which will install an openMosix kernel source tree in /usr/src/linux along with the openMosix userland tools.

Michael Imhof, aka tantive, keeps Gentoo current for the latest openMosix version.

Daniel Robbins, the President/CEO of Gentoo Technologies, Inc. and the creator of Gentoo Linux, wrote the artitles we use as our Introduction to openMosix Clusters.

4.8. Other distributions

Based on the explanations above you should be able to install openMosix on most other Linux platforms.

5.1. Easy Configuration

The auto-discovery daemon (omdiscd) provides a way to automatically configure an openMosix cluster hence eliminating the need of a /etc/mosix.map or similar manual configurations. Auto-discovery uses multicast packages to notify other nodes that it is an openMosix node. This way adding an extra node to your mosix cluster means that you just have to start the omdiscd on your machine and it will join the cluster.

However there are some small requirements, Like with any openMosix cluster , you need to have networking configured correctly. mainly the routing. Without a default route, you must specify an interface to omdiscd with the -i option. Otherwise omdiscd will exit with an error like.

Aug 31 20:41:49 localhost omdiscd[1290]: Unable to determine address of default interface. This may happen because there is no default route configured. Without a default route, an interface must be: Network is unreachable Aug 31 20:41:49 localhost omdiscd[1290]: Unable to initialize network. Exiting.

An example of a correct routing is below

[root@localhost log]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth0 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo 0.0.0.0 10.0.0.99 0.0.0.0 UG 0 0 0 eth0

Basically from now on everything will get easier. Just start

omdiscd

And have a look at your logfiles you should see something similar to this

Sep 2 10:00:49 oscar0 kernel: openMosix configuration changed: This is openMosix #2780 (of 6 configured) Sep 2 10:00:49 oscar0 kernel: openMosix #2780 is at IP address 192.168.10.220 Sep 2 10:00:49 oscar0 kernel: openMosix #2638 is at IP address 192.168.10.78 Sep 2 10:00:49 oscar0 kernel: openMosix #2646 is at IP address 192.168.10.86 Sep 2 10:00:49 oscar0 kernel: openMosix #2627 is at IP address 192.168.10.67 Sep 2 10:00:49 oscar0 kernel: openMosix #2634 is at IP address 192.168.10.74

Congratulations , your openMosix cluster is now working.

omdiscd has some other options that you can use. You can either run omdiscd as a daemon (default) or in the foreground where output goes to the screen (standard output) omdiscd -n . An interface can be specified with the -i option.

Now lets still have a short look at the other tool , it's showmap. This tool will show you the newly auto generated openMosix map.

[root@oscar0 root]# showmap My Node-Id: 0x0adc Base Node-Id Address Count ------------ ---------------- ----- 0x0adc 192.168.10.220 1 0x0a4e 192.168.10.78 1 0x0a56 192.168.10.86 1 0x0a43 192.168.10.67 1 0x0a4a 192.168.10.74 1

Auto-discovery has some other features not listed here such as a routing mechanism for clusters with more than one network. More detailed information can be found in the README and DESIGN files in the user-land tools source tree.

More recent versions of the openMosix rc scripts will first verify wether an /etc/openmosix.map file or similar exists before trying to use autoconfiguration.

5.2. Compiling auto-discovering

If you are compiling autodiscovery from source you will need to make a small modification to openmosix.c. One of the first lines will be

#define ALPHA

You will need to put this in comment. If you want to have some more logging available to you you should edit main.c to show log_set_debug(DEBUG_TRACE_ALL); (somewhere around line 84) now run

% make clean % make

5.3. Troubleshooting autodiscovery

Sometimes however autodiscovery does not function as you would like, for example a node might not see multicast traffic from other nodes. This has occurred with some PCMCIA ethernet drivers. One solution is to place the interface in promiscuous and or multicast mode as detailed below:

Aug 31 20:45:58 localhost kernel: openMosix configuration changed: This is openMosix #98 (of 1 configured) Aug 31 20:45:58 localhost kernel: openMosix #98 is at IP address 10.0.0.98Aug 31 20:45:58 localhost omdiscd[1627]: Notified kernel to activate openMosix Aug 31 20:45:58 localhost kernel: Received an unauthorized information request from 10.0.0.99

What you should to then is try to force your NIC into promiscuous and/ or multicast mode manually.

ifconfig ethx promisc or ifconfig ethx multicast

You might also want to run

tcpdump -i eth0 ether multicast

which will have the same effect but you will now also be able to see the packages yourself.

Aug 31 22:14:43 inspon omdiscd[1422]: Simulated notification to activate openMosix [root@inspon root]# showmap My Node-Id: 0x0063 Base Node-Id Address Count ------------ ---------------- ----- 0x0063 10.0.0.99 1 [root@inspon root]# /etc/init.d/openmosix status OpenMosix is currently disabled [root@inspon root]#

If you see the simulated you have probably forgotten to put the

#define ALPHA

in comment.

I have also noticed that autodiscovery does not work with FireWire based network cards.

6.1. Cluster Installations

This chapter does not deal with installing openMosix as such, it does however deal with installing multiple machines with openMosix. Automated or semi automated mass installs.

6.2. DSH, Distributed Shell

At the time of this writing (May 2003) DSH's most current release is available from http://www.netfort.gr.jp/~dancer/software/downloads/ More info on the package can be found on http://www.netfort.gr.jp/~dancer/software/dsh.html The latest version available for download is 0.23.6 You will need both libdshconfig-0.20.8.tar.gz and dsh-0.23.5.tar.gz Start with installing libdshconfig

./configure make make install

Repeat the process for the dsh package.

Say we have a small cluster with a couple of nodes. To make life easier we want type each command once but have it executed on each node. You then have to create a file in $HOME/.dsh/group/clusterwname that lists the ip's of your cluster. eg.

[root@inspon root]# cat .dsh/group/mosix 192.168.10.220 192.168.10.84

As an example we run ls on each of these machines We use -g to use the mosix group (this way you can create subsets of a group with different configurations)

[root@inspon root]# dsh -r ssh -g mosix ls 192.168.10.84: anaconda-ks.cfg 192.168.10.84: id_rsa.pub 192.168.10.84: install.log 192.168.10.84: install.log.syslog 192.168.10.84: openmosix-kernel-2.4.17-openmosix1.i686.rpm 192.168.10.84: openmosix-tools-0.2.0-1.i386.rpm 192.168.10.220: anaconda-ks.cfg 192.168.10.220: id_dsa.pub 192.168.10.220: id_rsa.pub 192.168.10.220: openmosix-kernel-2.4.17-openmosix1.i686.rpm 192.168.10.220: openmosix-tools-0.2.0-1.i386.rpm 192.168.10.220: oscar-1.2.1rh72 192.168.10.220: oscar-1.2.1rh72.tar.gz

Note that neither of the machines ask for a password. This is because we have set up rsa authentication between the different accounts. If you want to run commands with multiple parameters you will have either have to put the command between quotes.

[root@inspon root]# dsh -r ssh -g mosix "uname -a" 192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown 192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown

or use the -c -- option. Both give basically the same output.

[root@inspon root]# dsh -r ssh -g mosix -c -- uname -a 192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown 192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown

7.1. Basic Administration

openMosix provides the advantage of process migration to HPC-applications. The administrator can configure and tune the openMosix-cluster by using the openMosix-user-space-tools or the /proc/hpc interface which will be now described in detail.

Up till openMosix version 2.4.16 the /proc interface was named /proc/mosix ! Until openMosix version 2.4.17 it was named /proc/hpc.

7.2. Configuration

The values in the flat files in the /proc/hpc/admin directory presenting the current configuration of the cluster. Also the administrator can write its own values into these files to change the configuration during runtime, e.g.

...

7.3. the userspace-tools

These following tools are providing easy administration to openMosix clusters.

migrate -send a migrate request to a process syntax: migrate [PID] [openMosix_ID]

mon -is a ncurses-based terminal monitor several informations about the current status are displayed in bar-charts

mosctl -is the openMosix main configuration utility syntax: mosctl [stay|nostay] [lstay|nolstay] [block|noblock] [quiet|noquiet] [nomfs|mfs] [expel|bring] [gettune|getyard|getdecay] mosctl whois [openMosix_ID|IP-address|hostname] mosctl [getload|getspeed|status|isup|getmem|getfree|getutil] [openMosix_ID] mosctl setyard [Processor-Type|openMosix_ID||this] mosctl setspeed interger-value mosctl setdecay interval [slow fast]

mosrun          -run a special configured command on a chosen node
                syntax:
                        mosrun  [-h|openMosix_ID| list_of_openMosix_IDs] command [arguments]

The mosrun command can be executed with several more commandline options. To ease this up there are several preconfigured run-scripts for executing jobs with a special (openMosix) configuration.

setpe           -manual node configuration utility
                syntax:
                        setpe   -w -f   [hpc_map]
                        setpe   -r [-f  [hpc_map]]
                        setpe   -off

-w reads the openMosix configuration from a file (typically /etc/hpc.map)
-r writes the current openMosix configuration to a file (typically /etc/hpc.map)
-off turns the current openMosix configuration off

tune            openMosix calibration and optimizations utility.
                (for further informations review the tune-man page)

Additional to the /proc interface and the commandline-openMosix utilities (which are using the /proc interface) there is a patched "ps" and "top" available (they are called "mps" and "mtop") which displays also the openMosix-node ID on a column. This is useful for finding out where a specific process is currently being computed.

This actually summarised the command line tools, but have a look at openMosixview which is a GUI for the most common administration tasks, and which ill be discussed in a future chapter.

7.4. Cluster Mask

(by Moshe Bar)

Several people have asked for a feature in openMosix which allows to specifiy to which nodes a given process and it's children can migrate and to which nodes it cannot.

Simone Ettore has just committed a new patch to the CVS which allows you to do just that.

Here is how it works:

• /proc/[pid]/migfilter enable/disable the capability of filter migration.

• /proc/[pid]/mignodes is a bit-list of nodes. The bit position of a node is calculated as 2^(PE-1). PE is node number.

• /proc/[pid]/migpolicy is the policy of the filtering: 0=DENY: the process can migrate in all nodes except when the relative bit on mignodes is 1 1=ALLOW: the process can migrate in all nodes where the relative bit on mignodes is 1

We are shortly going to release also a simple user-land tool to set the node mask, but I would like you guys to give it a try asap before we release it as openMosix 2.4.20-3.

8.1. Introduction

Some of the parts below are still from the old Mosix Howto, as time passes these parts will get replaced by relevant openMosix parts, however some things are still the same , but your mileage may vary.

8.2. Creating a "Master" node

Although openMosix architcture does not require a master node as such, you might want to have a head node from where you launch processes, this might be a multihomed node from where users log in to your cluster. You want to configure your machine to make processes migrate away

You have to trick the node in thinking it is the slowest node around and it'd better migrate all it's processes to the faster nodes.

You will have to make it "slow" with :

mosctl setspeed [n]

where n should be much lower than the speed of the other nodes Processes will move/migrate away fast. You can get the speed of a node with :

mosctl getspeed

8.3. Optimizing Mosix

Editorial Comment: To be checked with openMosix versions

Login a normal terminal as root. Type

setpe -r

which, if everything went right, will give you a listing of your /etc/mosix.map. If things did not go right, try

setpe -w -f /etc/mosix.map

to set up your node. Then, type

cat /proc/$$/lock

to see if your child processes are locked in your mode (1) or can migrate (0). If for some reason you find your processes are locked, you can change this with

echo 0 > /proc/$$/lock

until you fix the problem. Repeat the whole configuration scheme for a second computer. The programs tune_kernel and prep_tune that Mosix uses to calibrate the individual nodes do not work with the SuSE distribution. However, you can fake it. First, bring the computer you want to tune and another computer with Mosix installed down to single user mode by typing

init 1

as root. All other computers on the network should be shutdown if possible. On both machines, run the following commands:

/etc/init.d/network start /etc/init.d/mosix start echo 1 > /proc/mosix/admin/quiet

This fakes prep_tune and the first parts of tune_kernel. Note that if you have a laptop with a pcmcia network card, you will have to run

/etc/init.d/pcmcia start

instead of "/etc/init.d/network start". On the computer you want to tune, run tune_kernel and follow instructions. Depending on your machines, this can take a while - if you have a dog, this might be the time to go on that long, long walk you've always promised him. tune_kernel will create a program called "pg" in /root for testing reasons. Ignore it. After tuning is over, copy the contents of /tmp/overheads to the file /etc/overheads (and/or recompile the kernel). Repeat the tuning procedure for each computer. Reboot, enjoy Mosix, and don't forget to brag to your friends about your new cluster.

8.4. Channel Bonding Made Easy

Contributed by Evan Hisey

Channel bonding is actually horrible easy. This may explain the lack of documentation on this subject A bonded network appears as a normal network to the applications. All machines on a subnet must be either bonded the same way. Bonded and non-bonded machine really don't talk well to each other.

Channel bonding needs at least to physical sub-nets but can have more(Currently I have a tri-bonded cluster). To enable bonding you need to either compile in to the kernel or as a module (bonding.o) the Channel Bonding kernel code, as of 2.4.x is it a standard option of the kernel. The NIC's are setup as normal with except that you only us 'ifconfig' to initialize the first card of the bond. 'ifenslave' is used to initialize the remaining cards in the bonded connection. 'ifenslave' can be locate in the linux/Documentation/network/ directory. It will need to be compiled as it is a .c file. The basic format for use is

ifenslave <master> <slave1> <slave2>

...'. Channel bonded networks can connect to standard networks via a router or bridge that supports channel bonding( I just use an extra NIC and port-forwarding in the head node).

8.5. Updatedb

Updatedb in combination with mfs can cause some issues, you might want to add /mfs to the PRUNEFPATHS or mfs to the PRUNEFS in your /etc/updatedb.conf to disable updatedb from indexing this mountpoints.

8.6. openMosix and FireWire

openMosix does gain performance by using another type of network device, as described within the paper about openMosix and FireWire

9.1. Introduction

openMosixview is the next version and a complete rewrite of Mosixview. It is a cluster-management GUI for openMosix-cluster and everybody is invited to download and use it (at your own risk and responsibility). The openMosixview-suite contains 5 useful applications for monitoring and administrating openMosix-cluster.

All parts are accessible from the main application window. The most common openMosix-commands are executable by a few mouse-clicks. An advanced execution dialog helps to start applications on the cluster. "Priority-sliders" for each node simplifying the manual and automatic load-balancing. openMosixview is now adapted to the openMosix-auto-discovery and gets all configuration-values from the openMosix /proc-interface.

9.2. openMosixview vs Mosixview

openMosixview is fully designed for openMosix cluster only. The Mosixview-website (and all mirrors) will stay as they are but all further developing will continue with openMosixview located at the new domain www.openmosixview.com

If you have: questions, features wanted, problems during installation, comments, exchange of experiences etc. feel free to mail me, Matt Rechenburg or subscribe to the openMosix/Mosixview-mailing-list and mail to the openMosix/Mosixview-mailing-list

changes: (to Mosixview 1.1) openMosixview is a complete rewrite "from the scratch" of Mosixview! It has the same functionalities but there are fundamental changes in ALL parts of the openMosixview source-code. It is tested with a constantly changing cluster topography (required for the openMosix auto-discovery) All "buggy" parts are removed or rewritten and it (should ;) run much more stable now.

9.3. Installation

Requirements

Documentation about openMosixview There is a full HTML-documentation about openMosixview included in every package. You find the startpage of the documentation in your openMosixview installation directory: openmosixview/openmosixview/docs/en/index.html

The RPM-packages have their installation directories in: /usr/local/openmosixview

9.4. using openMosixview

9.5. openMosixprocs

9.6. openMosixcollector

The openMosixcollector is a daemon which should/could be started on one cluster-member. It logs the openMosix-load of each node to the directory /tmp/openmosixcollector/* These history log-files analyzed by the openMosixanalyzer (as described later) gives an nonstop overview of the load, memory and processes in your cluster. There is one main log-file called /tmp/openmosixcollector/cluster Additional to this there are additional files in this directory to which the data is written.

At startup the openMosixcollector writes its PID (process id) to /var/run/openMosixcollector.pid

The openMosixcollector-daemon restarts every 12 hours and saves the current history to /tmp/openmosixcollector[date]/* These backups are done automatically but you can also trigger this manual.

There is an option to write a checkpoint to the history. These checkpoints are graphically marked as a blue vertical line if you analyze the history log-files with the openMosixanalyzer. For example you can set a checkpoint when you start a job on your cluster and another one at the end..

Here is the explanation of the possible commandline-arguments:

openmosixcollector -d //starts the collector as a daemon openmosixcollector -k //stops the collector openmosixcollector -n //writes a checkpoint to the history openmosixcollector -r //saves the current history and starts a new one openmosixcollector //print out a short help

You can start this daemon with its init-script in /etc/init.d or /etc/rc.d/init.d. You just have to create a symbolic link to one of the runlevels for automatic startup.

How to analyze the created logfiles is described in the openMosixanalyzer-section.

9.7. openMosixanalyzer

9.8. openMosixmigmon

9.9. openmosixview FAQ

9.10. openMosixview + ssh:

(this HowTo is for SSH2) You can read the reasons why you should use SSH instead of RSH everyday on the newspaper when another script-kiddy hacked into an insecure system/network. So SSH is a good decision at all.

freedom x security = constant (from a security newsgroup)

That is why it is a bit tricky to configure SSH. SSH is secure even if you use it to login without being prompted for a password. Here is a (one) way to configure it.

At first a running secure-shell daemon on the remote site is required. If it is not already installed install it! (rpm -i [sshd_rpm_packeage_from_your_linux_distribution_cd]) If it is not already running start it with:

/etc/init.d/ssh start

Now you have to generate a keypair for SSH on your local computer whith ssh-keygen.

ssh-keygen

You will be prompt for a passphrase for that keypair. The passphrase normally is longer than a password and may be a whole sentence. The keypair is encrypted with that passphrase and saved in

/root/.ssh/identity //your private key and /root/.ssh/identity.pub //your public key

Do NOT give your private-key to anybody!!! Now copy the whole content of /root/.ssh/identity.pub (your public-key which should be one long line) into /root/.ssh/authorized_keys on the remote host. (also copy the content of /root/.ssh/identity.pub to your local /root/.ssh/authorized_keys like you did it with the remote-node because openMosixview needed password-less login to the local-node too!)

If you ssh to this remote host now you will be prompted for the passphrase of your public-key. Giving the right passphrase should give you a login.

What is the advantage right now??? The passphrase is normally a lot longer than a password!

The advantage you can get using the ssh-agent. It manages the passphrase during ssh login.

ssh-agent

The ssh-agent is started now and gives you two environment-variables you should set (if not set already). Type:

echo $SSH_AUTH_SOCK and echo $SSH_AGENT_PID

to see if they are exported to your shell right now. If not just cut and paste from your terminal. e.g. for the bash-shell:

SSH_AUTH_SOCK=/tmp/ssh-XXYqbMRe/agent.1065 export SSH_AUTH_SOCK SSH_AGENT_PID=1066 export SSH_AGENT_PID

example for the csh-shell:

setenv SSH_AUTH_SOCK /tmp/ssh-XXYqbMRe/agent.1065 setenv SSH_AGENT_PID 1066

With these variables the remote-sshd-daemon can connect your local ssh-agent by using the socket-file in /tmp (in this example /tmp/ssh-XXYqbMRe/agent.1065). The ssh-agent can now give the passphrase to the remote host by using this socket (it is of course an encrypted transfer)!

You just have to add your public-key to the ssh-agent with the ssh-add command.

ssh-add

Now you should be able to login using ssh to the remote host without being prompted for a passwod!

You could (should) add the ssh-agent and ssh-add commands in your login-profile e.g.

eval `ssh-agent` ssh-add

Now it is started when you login on your local workstation. You have done it! I wish you secure logins now.

openMosixview There is a menu-entry which toggles using rsh/ssh with openMosixview. Just enable this and you can use openMosixview even in insecure network-environments. You should also save this configuration (the possibility for saveing the current config in openMosixview was added in the 0.7 version) because it gets initial data from the slave using rsh or ssh (just like you configured).

If you choose a service wich is not installed properly openMosixview will not work! (e.g. if you cannot rsh to a slave without being prompted for a password you cannot use openMosixview with RSH; if you cannot ssh to a slave without being prompted for a password you cannot use openMosixview with SSH)

10.1. Introduction

There are a couple of different applications available to monitor and admin openMosix, we give a short overview of them in this chapter, we won't really go in detail.

10.2. openMosixView

openMosixview is the most used and the best known applet for openMosix administration, you can read more about it in the openMosix adminstration chapter.

10.3. openMosixapplet

The openMosixApplet lets you watch the realtime load of your openMosix cluster. It consists of a local daemon which listens for connections by applets. The applet uses chart2D to provide a good-lookin' feeling.

10.4. wmonload

wmomload is a simple, but handy and small dockapp for overviewing the load of cluster nodes in a small openMosix-based cluster.

10.5. openMosixWebView

openMosixWebView - Produces web charts for monitoring an openMosix cluster. openMosixWebView is a PHP script for monitoring an openMosix cluster via the WEB. It uses openMosixview's openMosixCollector logs. Download now the last release openmosixwebview-0.2.12.tar.gz (16 Feb 2003)

See openMosixWebView screenshots and running :-) Released under the GNU General Public License (GPL). See README and FAQ files.

11.1. Introduction

Although most of the issues in this chapter could be a part of the FAQ. There where the FAQ will give a short "how to solve" answer, we have taken a closer look at them and explained why they are problems and how to solve them.

11.2. My processes won't migrate

Help process XYZ doesn't migrate. Moshe Bar explains below why some processes migrate and why some don't. But before that you can always look in /proc/$pid/, there often is a cantmove file which will tell you why a certain process can't migrate.

Processes can also be locked. You can check if a process is locked with:

cat /proc/$PID/lock

where $PID is the processid of the process in question.

Now listen to what Moshe himself has to say about this topic.

Often people have the same kernel but on a different distribution, say a mixed environment of RedHat and Debian ,rc scripts from different distros tend to start openmosix differently. Some implementations completely modify /etc/inittab to start all daemons (and their children) with

mosrun -h

. So that they won't migrate. Therefore all these processes have a 1 in /proc/pid/lock when you start. You can force them to migrate by writing a 0 to this file .

Ok, this simple program should always migrate if launched more times than number of local CPUs. So for a 2-way SMP system, starting this program 3 times will start migration if the other nodes in the cluster have at least the same speed like the local ones:

int main() { unsigned int i; while (1) { i++; } return 0; }

On a Pentium 800Mhz CPU it takes quite a while to overflow.

This sample program with content like this will never migrate:

#include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h> ... key_t key; /* key to be passed to shmget() */ int shmflg; /* shmflg to be passed to shmget() */ int shmid; /* return value from shmget() */ int size; /* size to be passed to shmget() */ ... key = ... size = ... shmflg) = ... if ((shmid = shmget (key, size, shmflg)) == -1) { perror("shmget: shmget failed"); exit(1); } else { (void) fprintf(stderr, "shmget: shmget returned %d\n", shmid); exit(0); } ...

Program using pipes like this do migrate nicely:

int pdes[2]; pipe(pdes); if ( fork() == 0 ) { /* child */ close(pdes[1]); /* not required */ read( pdes[0]); /* read from parent */ ..... } else { close(pdes[0]); /* not required */ write( pdes[1]); /* write to child */ ..... }

MODIFIED Programs using pthreads since version 2.4.17 don't migrate, however they don't segfault anymore.

// // Very simple program demonstrating the use of threads. // // Command-line argument is P (number of threads). // // Each thread writes "hello" message to standard output, with // no attempt to synchronize. Output will likely be garbled. // #include <iostream> #include <cstdlib> // has exit(), etc. #include <unistd.h> // has usleep() #include <pthread.h> // has pthread_ routines // declaration for function to be executed by each thread void * printHello(void * threadArg); // ---- Main program ------------------------------------------------- int main(int argc, char* argv[]) { if (argc < 2) { cerr << "Usage: " << argv[0] << " numThreads\n"; exit(EXIT_FAILURE); } int P = atoi(argv[1]); // Set up IDs for threads (need a separate variable for each // since they're shared among threads). int * threadIDs = new int[P]; for (int i = 0; i < P; ++i) threadIDs[i] = i; // Start P new threads, each with different ID. pthread_t * threads = new pthread_t[P]; for (int i = 0; i < P; ++i) pthread_create(&threads[i], NULL, printHello, (void *) &threadIDs[i]); // Wait for all threads to complete. for (int i = 0; i < P; ++i) pthread_join(threads[i], NULL); // Clean up and exit. delete [] threadIDs; delete [] threads; cout << "That's all, folks!\n"; return EXIT_SUCCESS; } // ---- Code to be executed by each thread --------------------------- // pre: *threadArg is an integer "thread ID". // post: "hello" message printed to standard output. // return value is null pointer. void * printHello(void * threadArg) { int * myID = (int *) threadArg; cout << "hello, world, "; // pointless pause, included to make the effects of // synchronization (or lack thereof) more obvious usleep(10); cout << "from thread " << *myID << endl; pthread_exit((void* ) NULL); }

Programs using all kinds of file descriptors, including sockets do migrate (sockets are not migrated with the process however, files are migrated if using oMFS/DFSA)

(all above code is by Moshe as Moshe Bar or by Moshe as CTO of Qlusters, Inc.)

Please also refer to the man pages of openMosix , they also give an adequate explanation why processes don't migrate.

If for some reason your processes stay locked while they shouldn't. You can try to allow locked processes to migrate by simply putting

# tell shells to allow subprocs to migrate to other nodes echo 0 > /proc/self/lock

into "/etc/profile" Warning : this fix will allow all process to migrate not just the ones you want. To only allow specific process to migrate use 'mosrun -l' to unlock only the desired process.

11.3. I don't see all my nodes

First of all , are you using the same kernel version on each machine ? The 'same-kernel' refers to the version. You can build different kernel images of the same source version to meet the hardware/software needs of a given node. However you wil need toe make sure that when you install openMosix on your cluster, all your machines should have the openmosix-x.x.x-y kernel installed, in contrast to having one machine running openmosix-x.x.z-x, another running openmosix-x.x.x-y, another running openmosix x.x.x-z, and so on and so forth

When you run mosmon, press t to see the total of machines running. Does it warn you that mosix is not running?

If yes, then make sure your machine's ip is included in /etc/mosix.map (don't use 127.0.0.1 - if your machine's ip is such, then you probably have problems with your dhcp server/nameserver). If it does not tell you that mosix is not running, see what machines show up. Do you see only your machine?

If yes, then your machine is most likely running a firewall and is not letting openmosix through.

If not, then the problem is most likely with the machine that doesn't show up. Also: Do you have two nic cards on a node? then you have to edit the /etc/hosts file to have a line that has the following format

non-cluster_ip cluster-hostname.cluster-domain cluster-hostname

You might also need to set up a routing table, which is a whole different subject.

Maybe you used different kernel-parameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum network-topology complexity support' on each machine.

11.4. I often get errors: No such process

I often get the error

bash: child setpgid (4061 to 4061): No such process

what does this mean ?

The above line meas that the shell you were using has acutallly migrated to another node ? This printout from bash is caused by a bug in old version of openmosix, but a fix has been commited. (Muli Ben-Yehuda mulix@actcom.co.il)

11.5. DFSA ? MFS ?

People often get confused about what exactly MFS and DFSA are. As discussed before in the howto MFS is the feature of openMosix that enables you access to remote filesystems as if those filesystems were locally mounted. They are mostly mounted on /mfs . A common misunderstanding is that you need MFS in order to have openMosix working, this is not true, however it can make things easier.

With DFSA enabled, system calls will be executed on the remote node withouth migrating the process back to it's home node. This behaviour (direct filesystem access) causes processes migratiing to the data and not the other way around (which is common). If DFSA is not enabled MfS is "just" a non-caching network-filesystem.

Very generally speaking, if you don't have DFSA turned on, each and every I/O will go to the home node for execution. With DFSA turned on, if the file happens to be residing on the node where the process finds itself then the I/O will happen locally.

A very common error is that people mix kernels with DFSA enabled and disabled. So one has to have a way to find out wether DFSA is actually enabled. This information can be obtained by typing

cat /proc/hpc/admin/version

11.6. Python Troubles

Some people have reported problems with Python, closer research showed that these problems were not with openMosix but rather with glibc issues, however it seemed that issues manifested themselves especially in openMosix.

One user solved the problem by removig /lib/i686/lib* on his machine and let the applications link dynamically against /lib/libpthread (and other) However bugfixes in newer glibc versions combined with more recent openMosix version seem to have solved these problems.

12.1. Locked Processes

If for some reason you find your processes are always locked in your home node and you can't find the reason, you can put the following lines into your ~/.profile as a stop-gap measure to automatically enable migration:

if [ -x /proc/$$/lock ]; then echo 0 > /proc/$$/lock fi

However, you should make an effort to find out what the problems is

12.2. Choosing your processes

You will probably want to test your setup before deciding which programs you want to enable migration for. For example, if you are running KDE2 on a slow machine and have a significantly faster machine has part of your Mosix cluster, you might find resource-hungry programs like kmail are migrated out. This is not a bad thing as such, however, it can lead to a brief moment when your writing is not displayed on the screen immediately.

12.3. Java and openMosix

Green Threads JVM's, allow for migration because each Java thread is a separate process. Threads other than Java green thread JVM's cannot be migrated by Linux, so openMosix cannot migrate programs that use them.

If you have the source so your Java application you might be able to compile the application native. In this case you might be able to migrate your applications to another node.

Gian Paolo Ghilardi wrote a paper titled Consideration on OpenMosix it deals amongst other topics with Java an dopenMosix. http://www.democritos.it/events/openMosix/papers/crema.pdf

12.4. openMosix and Hyperthreading

Basically openMosix performance increases with the current Linux scheduler when Hypethreading is disabled. You can do this by either entering 'noht' as a boot option or disabling HT in the bios.

For those who are still wondering what hypetrheading is : Intel explains it

12.5. openMosix and Firewalls

People often have questions regarding openMosix and firewalls. Amit helped me out on this matter:

from hpc/comm.c: #define MIG_DAEMON_PORT 0x3412 #define INFO_DAEMON_PORT 0x3415

converted to decimal, they are: 4660 and 5428 resp. (convert 1234, and not 3412, 'cos of the network-host byte ordering conversions.. Read the IP/TCP/UDP RFCs for info)

the mig_daemon port is a tcp port, the info_daemon port is udp. Hence tcp/4660 and udp/5428, Matt also mentions tcp/723 somewhere.

13.1. A small Test Script

The fastest way to test your openMosix cluster is by creating a small script with the following content.

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' &

I've saved it as test_mosix, now when I want to see If everything works I start op mosmon and I launch this script for a zillion times as in

for i in `ls /etc/` ; do ./test_mosix ; done

Now watch openMosix kicking in after a few seconds ...

13.2. Perl Proggie by Charles Nadeau

Perl program to test an openMosix Cluster.

Here is a is quick program I wrote to test an openMosix cluster. This is taken from a posting I made to the openMosix-devel mailing list on March 6th, 2002: "Charles wrote this little program (in Perl) to stress test his home cluster (3 P200MMX and a P166). It is a program simulating different sets of stocks in a portfolio for a given period of time. The code is well documented and it should be easy to add/remove stocks and change the average monthly yield and standard deviation for each stock. Since the problem of portfolio optimization cannot be solved analytically, it simulate a lot of portfolios and report the results at the end. Please note that this program does not take stock correlation into account. It is not finished yet but it's a good start. I plan to add more code at the end of the program to improve the reporting format of the data (generating SVG graph on the fly). But the simulation part works very well. In order to take advantage of the parallelism offered by openMosix, it uses the Perl module Parallel::?ForkManager (from CPAN) to span threads that openMosix can then assign to all the machines of the cluster (it also require another module for the statistical calculations, don't forget to install both, I provide the URLs in the comments of the code). Take a look at it and tell me what you think. Cheers!"

#! /usr/bin/perl -w # this mill unlock this process and all tis childs sub unlock { open (OUTFILE,">/proc/self/lock") || die "Could not unlock myself!\n"; print OUTFILE "0"; } unlock; # this will count the nodes sub cpucount { $CLUSTERDIR="/proc/hpc/nodes/"; $howmany=0; opendir($nodes, $CLUSTERDIR); while(readdir($nodes)) { $howmany++; } $howmany--; $howmany--; closedir ($nodes); return $howmany; } my $processes=cpucount; $processes=$processes*3; print("starting $processes processes\n"); #Portfolio.pl, version 0.1 #Perl program that simulate a portfolios for various stock composition for a given period of time #We run various scenarios to find the mix of assets that give the best performance/risk ratio #This method is base on the book "The intelligent asset allocator" by William Bernstein #Can be used to test an OpenMosix cluster #This program is licensed under GPL #Author: Charles-E. Nadeau Ph.D., (c) 2002 #E-mail address: charlesnadeau AT hotmail DOT com use Parallel::ForkManager; #We use a module to parallelize the calculation #Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2589 use Statistics::Descriptive::Discrete; #A module providing statistical values #Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?new=Search;filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2988 srand; #We initialize the random number generator #Initializing constant $NumberOfSimulation=$processes; #Number of simulation to run $NumberOfMonth=100000; #Number of month for wich to run the simulation $NumberOfStock=6; #Number of different stocks in the simulation #Portfolio to simulate #TODO: Read the stock details from a file $Stock[0][0]="BRKB"; #Stock ticker $Stock[0][1]=0.01469184; #Stock average monthly return $Stock[0][2]=0.071724934; #Stock average monthly standard deviation $Stock[1][0]="TEST "; #Stock ticker $Stock[1][1]=-0.01519; #Stock average monthly return $Stock[1][2]=0.063773903; #Stock average monthly standard deviation $Stock[2][0]="SPDR"; #Stock ticker $Stock[2][1]=0.008922718; #Stock average monthly return $Stock[2][2]=0.041688404; #Stock average monthly standard deviation $Stock[3][0]="BRKB"; #Stock ticker $Stock[3][1]=0.01469184; #Stock average monthly return $Stock[3][2]=0.071724934; #Stock average monthly standard deviation $Stock[4][0]="TEST "; #Stock ticker $Stock[4][1]=-0.01519; #Stock average monthly return $Stock[4][2]=0.063773903; #Stock average monthly standard deviation $Stock[5][0]="SPDR"; #Stock ticker $Stock[5][1]=0.008922718; #Stock average monthly return $Stock[5][2]=0.041688404; #Stock average monthly standard deviation my $pm = new Parallel::ForkManager($NumberOfSimulation); #Specify the number of threads to span $pm->run_on_start( sub { my ($pid,$ident)=@_; print "started, pid: $pid\n"; } ); #We initialize the array that will contain the results @Results=(); for $i (0..$NumberOfSimulation-1){ for $j (0..$NumberOfStock+3){ $Results[$i][$j]=0.0; #Equal to 0.0 to start } } for $i (0..$NumberOfSimulation-1){ #Loop on the number of simulation to run $Results[$i][0]=$i; #The first column of each line is the number of the simulation $pm->start and next; #Start the thread $TotalRatio=1; #The sum of the proprtion of each stock #Here we calculate the portion of each investment in the portfolio for a given simulation for $j (0..$NumberOfStock-2){ #We loop on all stock until the second to last one #TODO: Replace rand by something from Math::TrulyRandom $Ratio[$j]=rand($TotalRatio); $Results[$i][$j+1]=$Ratio[$j]; #We store the ratio associated to this stock $TotalRatio=$TotalRatio-$Ratio[$j]; } $Ratio[$NumberOfStock-1]=$TotalRatio; #In order to have a total of the ratios equal to one, we set the last ratio to be the remainder $Results[$i][$NumberOfStock]=$Ratio[$NumberOfStock-1]; #We store the ratio associated to this stock. Special case for the last stock $InvestmentValue=1; #Initially the investment value is 1 time the initial capital amount. my $stats=new Statistics::Descriptive::Discrete; #We initialize the module used to calculate the means and standard deviations for $j (1..$NumberOfMonth){ #Loop on the number of months $MonthlyGrowth[$j]=0.0; #By how much did we grow this month. Initially, no growth yet. #We loop on each stock to find its monthly contribution to the yield for $k (0..$NumberOfStock-1){ $MonthlyGrowth[$j]=$MonthlyGrowth[$j]+($Ratio[$k]*((gaussian_rand()*$Stock[$k][2])+$Stock[$k][1])); #We had the growth for this stock to the stock already calculated for the preceding stocks } $stats->add_data($MonthlyGrowth[$j]); #Add the yield for this month so we can later on have the mean and standard deviation for this simulation $InvestmentValue=$InvestmentValue*(1+$MonthlyGrowth[$j]); #Value of the Investment after this month } $Results[$i][$NumberOfStock+1]=$stats->mean(); #Calculate the average monthly growth $Results[$i][$NumberOfStock+2]=$stats->standard_deviation(); #Calculate the standard deviation of the monthly growth $pm->finish; #Finish the thread } $pm->wait_all_children; #We wait until all threads are finished #Printing the results print "Simulation "; for $j (0..$NumberOfStock-1){ print "Ratio$Stock[$j][0] "; } print " Mean StdDev YieldRatio\n"; for $i (0..$NumberOfSimulation-1){ printf "%10d ", $Results[$i][0]; for $j (1..$NumberOfStock){ printf " %6.2f ",$Results[$i][$j]; } if($Results[$i][$NumberOfStock+2]!=0) { printf "%5.4f %5.4f %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], ($Results[$i][$NumberOfStock+1]/$Results[$i][$NumberOfStock+2]); } else { printf "%5.4f %5.4f %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], 0; } } #Subroutines #Subroutine to generate two numbers normally distributed #From "The Perl Cookbook", recipe 2.10 sub gaussian_rand { my ($u1, $u2); my $w; my ($g1, $g2); do { $u1=2*rand()-1; $u2=2*rand()-1; $w=$u1*$u1+$u2*$u2; } while ($w>=1 || $w==0); #There was an error in the recipe, I corrected it here $w=sqrt(-2*log($w)/$w); $g2=$u1*$w; $g1=$u2*$w; return wantarray ? ($g1,$g2) : $g1; }

13.3. the openMosix stress-test

by Matt Rechenburg

14.1. Introduction

This Section is a Work in Progress

Lots of people try to use openMosix as a kind of compile farm, ofthen they come back very disappointed. This chapter of the howto will try to explain in which cases your compilations will benefit from openMosix and how to improve your successlevel.

First of all you have to remember 1 thing. openMosix will not migrate all processes you start on your cluster, only the ones that will benefit from migration to another node. For compiling this means that a process has to last long enough. Kernel compiles typically consist of numerous short compiles, each of them not being long enough to acutally migrate.

15.1. Introduction

This Section is a Work in Progress

Computer Graphics have always been applications that required a lot of CPU power, this hasn't changed. Within this chapter I wil demonstrate with some practical examples how Computer Graphics can benefit from openMosix.

15.2. Povray

The Persistence of Vision Raytracer is a high-quality, totally free tool for creating stunning three-dimensional graphics.

Ray-tracing is a rendering technique that calculates an image of a scene by simulating the way rays of light travel in the real world. However it does its job backwards. In the real world, rays of light are emitted from a light source and illuminate objects. The light reflects off of the objects or passes through transparent objects. This reflected light hits our eyes or perhaps a camera lens. Because the vast majority of rays never hit an observer, it would take forever to trace a scene.

These kind of applications can be easily made parrallel by using pvmpovray. Pvmpovray expects to working on a Beowulf style cluster and spread it's load to other nodes using pvm. The openMosix way of doing this is the same, however we just do this on 1 machine and have openMosix do the load spreading work fo you !

A GREAT Howto on PVM Povray will show you how to setup PVMPovray. Below is a small summary.

$ cd pvmpov3_1g_2 $ tar xfz ../povuni_s.tgz $ tar xfz ../povuni_d.tgz $ ./inst-pvm Trying to apply the patch. Searching for rejected files $

Now compile with aimk ( which is a wrapper script provided by the pvm rpm , buth which probably won't be in your path.(some of the readers might remember aimk from other platforms / applications )

If you are on a RH 8.0 box id moved libpng and zlib to .notused .. This in order to prevent version issues .. with other libpng and zlib versions.

export PATH=$PATH:/usr/share/pvm3/lib export PVMROOt=/usr/share/pvm3

I then run aimk newunix. Then we start pvm and quit it. The daemon stays active..

And a last thing that is not known by a novice pvm user is that pvm does use its own paths, and you have to put pvmpov either in that path or launch it with the complete pathname.

[root@dhcp71 povray31]# /usr/local/bin/pvmpov -L /usr/src/povray/pvmpov3_1g_2/povray31/include/ +Iskyvase.pov +Oskyvase.tga +NT16 +NW64 +NH64 +v +w1024 +h768 Persistence of Vision(tm) Ray Tracer Version 3.1g.Linux.gcc This is an unofficial version compiled by: Jakob Flierl - PVMPOV Version 3.1g.2 The POV-Ray Team(tm) is not responsible for supporting this version. Copyright 1999 POV-Ray Team(tm) Never found section in file /usr/src/povray/pvmpov3_1g_2/povray31/include/. Initializing PVMPOV Spawning /usr/local/bin/pvmpov with 16 PVM tasks on 1 hosts... ...16 PVM tasks successfully spawned. Waiting up to 120s for first slave to start... Slave 0 successfully started. Parsing Options Input file: skyvase.pov (compatible to version 3.1) Remove bounds........On Split unions........Off Library paths: /usr/local/lib/povray31 /usr/local/lib/povray31/include Output Options Image resolution 1024 by 768 (rows 1 to 768, columns 1 to 1024). Output file: skyvase.tga, 24 bpp PNG Graphic display.....Off Mosaic preview......Off CPU usage histogram.Off Continued trace.....Off Allow interruption...On Pause when done.....Off Verbose messages.....On Tracing Options Quality: 9 Bounding boxes.......On Bounding threshold: 25 Light Buffer.........On Vista Buffer.........On Antialiasing........Off Radiosity...........Off Animation Options Clock value.... 0.000 (Animation off) PVM Options Block Width.... 64 Block Height... 64 PVM Tasks...... 16 PVM Nice....... 5 PVM Arch....... PVM Slave...... /usr/local/bin/pvmpov PVM WorkingDir. /usr/src/povray/pvmpov3_1g_2/povray31 Redirecting Options All Streams to console..........On Debug Stream to console.........On Fatal Stream to console.........On Render Stream to console........On Statistics Stream to console....On Warning Stream to console.......On Starting frame 0... Slave 1 at dhcp71.office.be.stone-it.com successfully started. Slave 2 at dhcp71.office.be.stone-it.com successfully started. Slave 3 at dhcp71.office.be.stone-it.com successfully started. Slave 4 at dhcp71.office.be.stone-it.com successfully started. Slave 5 at dhcp71.office.be.stone-it.com successfully started. Slave 6 at dhcp71.office.be.stone-it.com successfully started. Slave 7 at dhcp71.office.be.stone-it.com successfully started. Slave 8 at dhcp71.office.be.stone-it.com successfully started. Slave 9 at dhcp71.office.be.stone-it.com successfully started. Slave 10 at dhcp71.office.be.stone-it.com successfully started. Slave 11 at dhcp71.office.be.stone-it.com successfully started. Slave 12 at dhcp71.office.be.stone-it.com successfully started. Slave 13 at dhcp71.office.be.stone-it.com successfully started. Slave 14 at dhcp71.office.be.stone-it.com successfully started. Slave 15 at dhcp71.office.be.stone-it.com successfully started. 0:00:53 86.46 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (77%) 0:00:53 86.98 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (67%) Not using dhcp71.office.be.stone-it.com for reassignment (86%) Not using dhcp71.office.be.stone-it.com for reassignment (85%) 0:00:55 89.06 of blocks complete. 640 of 768 lines finished (in frame 0).Not using dhcp71.office.be.stone-it.com for reassignment (65%) 0:00:56 91.67 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (72%) 0:00:56 92.71 of blocks complete.Not using dhcp71.office.be.stone-it.com for reassignment (80%) 0:00:57 93.75 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:57 94.79 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:58 95.83 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:58 96.35 of blocks complete. 672 of 768 lines finished (in frame 0).Not using dhcp71.office.be.stone-it.com for reassignment (77%) Slave at dhcp71.office.be.stone-it.com has exited. 0:00:58 97.14 of blocks complete. 688 of 768 lines finished (in frame 0). Slave at dhcp71.office.be.stone-it.com has exited. 0:00:59 97.92 of blocks complete. Slave at dhcp71.office.be.stone-it.com has exited. 0:00:60 98.44 of blocks complete. 704 of 768 lines finished (in frame 0). Slave at dhcp71.office.be.stone-it.com has exited. 0:01:03 100.00 of blocks complete. 768 of 768 lines finished (in frame 0). Finishing frame 0...rtw. 768 Waiting for remaining slave stats. PVM Task Distribution Statistics: host name [ done ] [ late ] host name [ done ] [ late ] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 7.81%] [ 0.07%] dhcp71.office.be.stone-it.com [ 8.85%] [ 1.17%]dhcp71.office.be.stone-it.com [ 4.69%] [ 0.00%] dhcp71.office.be.stone-it.com [ 8.85%] [ 0.98%]dhcp71.office.be.stone-it.com [ 4.17%] [ 0.00%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 8.33%] [ 0.52%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 5.73%] [ 0.72%] dhcp71.office.be.stone-it.com [ 7.29%] [ 2.73%]dhcp71.office.be.stone-it.com [ 4.17%] [ 0.00%] dhcp71.office.be.stone-it.com [ 5.21%] [ 0.00%]dhcp71.office.be.stone-it.com [ 6.77%] [ 0.13%] dhcp71.office.be.stone-it.com [ 4.69%] [ 0.00%]dhcp71.office.be.stone-it.com [ 7.81%] [ 0.00%] POV-Ray statistics for finished frames: skyvase.pov Statistics (Partial Image Rendered), Resolution 1024 x 768 ---------------------------------------------------------------------------- Pixels: 303104 Samples: 303104 Smpls/Pxl: 1.00 Rays: 1192710 Saved: 0 Max Level: 0/5 ---------------------------------------------------------------------------- Ray->Shape Intersection Tests Succeeded Percentage ---------------------------------------------------------------------------- Cone/Cylinder 1842227 900504 48.88 CSG Intersection 2742731 323346 11.79 CSG Union 1801008 521692 28.97 Plane 20223278 11233348 55.55 Quadric 1801008 1196533 66.44 Sphere 1801008 461786 25.64 Bounding Object 1842227 900504 48.88 ---------------------------------------------------------------------------- Calls to Noise: 1201944 Calls to DNoise: 2108954 ---------------------------------------------------------------------------- Shadow Ray Tests: 2856188 Succeeded: 85620 Reflected Rays: 889606 ---------------------------------------------------------------------------- Smallest Alloc: 9 bytes Largest: 20508 Peak memory used: 5643343 bytes ---------------------------------------------------------------------------- Time For Trace: 0 hours 1 minutes 7.0 seconds (67 seconds) Total Time: 0 hours 1 minutes 7.0 seconds (67 seconds)

As you can see the application was splitted into differen parts and run separatly, openMosix then did the job of balancing the load to other machines.

I had good results with 2 to 3 times the the number of cpu's I had available

16.1. Introduction

16.2. Blast

One of the more frequent used application in this field is Blast, Blast has a patch available that makes it work smoother with openMosix, but that's not the only alternative.

First of all there are some known problems with this patch, and other versions of blast , blast tends to segfault sometimes, this mostly happens with the preformatted databases you download from the internet. If you run formatdb on a raw database these errors tend to go away.

Next to the openMosix blast patch a lot of people run MPIBlast Given the fact that openMosix tends to speed up MPI, adding openMosix to this config might even give you more power for your money, however we will have to do some extra research to be able to confirm this.

17.1. Introduction

this part has been written by Amit Shah

There's not much documentation available right now for the kernel. I hope to write some in the coming weeks. Anyways, here's how the sources are laid out:

The openMosix code resides largely in hpc/ and include/hpc. There are lots of patches to the core kernel files everywhere, right from the arch/i386 directories to mm/, fs/, etc. You need to read up the code which interests you and think that would matter for the present situation (that shouldn't be a problem, since you've done kernel coding).

here's what you should expect in each of the source files:

• hpc/badops.c: one file to handle all the bad operations: mostly return err codes

• hpc/balance.c: The load balancer code (load + mem usage + n/w usage)

• hpc/comm.c: The intra-cluster communication setup

• hpc/config.c: The config code for openMosix: after you run the startup script

• hpc/decay.c: decay (age) the stats and info collected from other nodes

• hpc/deputy.c: Code executed on the deputy: service remote syscalls (ie. after the process has migrated), signals, etc.

• hpc/dfsa.c: Direct File System Access code: the distributed file system abstraction layer

• hpc/div.c: the algorithms to do floating point divisions

• hpc/export.c: export symbols needed in other files

• hpc/freemem.c: to keep track of free, avl. memory and to free it if need be. hugely taken from the Linux mm/ code.

• hpc/hpcadmin.c: tune openMosix admin values (through /proc/hpc)

• hpc/hpcproc.c: The /proc/hpc code is handled here

• hpc/info.c: The info daemon: sends and receives (multicast) load+mem usage stats throughout the cluster

• hpc/init.c: Initialization code: initializes the daemons, etc.

• hpc/kernel.c: most of the "core" code: all the important algorithms, decisions, etc. made here.

• hpc/load.c: calculation of local load, etc.

• hpc/mig.c: Code that handles the migration. Code in this file is invoked on any migration: deputy->remote, remote->deputy; remote->remote

• hpc/prequest.c: handles the process's requests: signals, more memory, etc.

• hpc/remote.c: Code executed when the process is on the remote: syscalls handling on remote, passing control to deputy, etc.

• hpc/rinode.c: fs/ related stuff: used mostly for DFSA

• hpc/service.c: setting up daemons, getting memory, etc.

• hpc/syscalls.c: handles all the remote syscalls here

• hpc/ucache.c: handles the ucache: mostly mm/, fs/ stuff.