• Home
• Download
• News

Learn More

• Introduction
• Features
• Documentation

Support

• Mailing List
• Cookbook
• Need more help?

Development

• ChangeLog
• RoadMap
• Stories
• Get Involved
• Source Code

Administration

• Sourceforge.net

Search

1.  Why the development of a MPI version of simuPOP was discontinued? (Mar. 10, 2009)

A prototype of a MPI version of simuPOP was added to simuPOP 0.7.5 in Dec. 2006. The general idea worked and I was even able to run small scripts using it. However, because a MPI version could not achieve its initial design goals, and because a full implementation required major revision to the simuPOP core, the MPI code was removed in simuPOP 0.8.3.

When the MPI version of simuPOP was first designed, 2G of RAM sounded huge and 64 bit operating systems were rare. A MPI version seemed to be the only way to break the 4G RAM barrier of 32 bit operating systems. I also hoped that a MPI version could significantly improve the performance of large simulations.

However, compare to other single-executable programs, a MPI implementation of simuPOP was extremely difficult to design. Because simuPOP is a programming environment, arbitrary user logic could be used. For example, a user could change genotype of a random individual using the Python random module. Different individuals could be chosen if the script is executed separately on different nodes, and lead to erroneous results. The only feasible MPI design would be a master-slave model where a master node interprets a script, and sends very detailed instructions to the slave nodes. However, this model requires a large amount of communication between nodes, especially with population changes. Consequently, the MPI modules may not provide any performance advantage over a regular module. This was more or less confirmed using my experimental implementation.

And you know what happened next. RAM became cheaper and cheaper and even home computers got 4G or more RAM. Dual-core or quad-core machines became commonplace and 64 bit operating systems became mainstream. Because it became easy to simulate large populations on a regular workstation, there was less and less a need for the MPI version of simuPOP.

Another reason for the removal of the MPI version is because I am looking into an openMP implementation. Using a shared-memory architecture, I might be able to simulate several replicates, or produce multiple offspring simultaneously using different threads. The performance boost could be dramatic. In addition, this implementation requires little modification to the simuPOP codebase and it is possible that I can distribute simuPOP modules that can run on both single and multi-core machines...

If everything moves as planned, simuPOP 1.0.x will be bug fix releases of simuPOP 1.0, and the 1.1.x releases will have openMP support.

2.  Icc vs. Gcc: which one is faster for simuPOP simulations? (Feb, 21, 2009)

simuPOP uses Visual C++ 2003 (win32, Python 2.4 and 2.5), Visual C++ Express 2008 (win32, Python 2.6), and GCC/G++ for all other platforms (MacOS, Linux, Solaris). These compilers are chosen because they are the compilers used for the official Python distributions.

Intel icc is usually considered to generate (much) faster code than gcc. I tried icc before, using a simuPOP version around 0.6.0 so I am interested to see whether or not simuPOP 0.9.2 can be compiled with icc.

Here are the steps:

• Download icc from Intel ICC website. The linux non-commercial version is free, and is the version I use.
• Install icc. I uses a separate user account and install icc locally to that user so that it will not mess up with my current development environment. I use csh and set a ~/.cshrc file as follows:

source /my/home/intel/Compiler/11.0/081/bin/iccvars.csh intel64
setenv PATH /my/home/Python26/bin:${PATH}

• Download Python 2.6 source code, and compile as follows

> tar zxf Python-2.6.tgz
> cd Python-2.6
> setenv CC icc
> setenv CXX icc
> ./configure --prefix=/my/home/Python26
> make
> make install

• (Optinal) download and install scons.
• Then, I check up a clean copy of simuPOP, and compile simuPOP as usual:

  > python setup.py install
  

> scons install

There are a lot more remarks (warnings) than gcc and many of them do not make much sense to me. Anyway, simuPOP is compiled successfully. All simuPOP tests and examples in the simuPOP user's guide run smoothly.

Does icc really help the performance of simuPOP? I do not have time for a thorough test so I run a typical random mating test in test_21_performance.

> cd simuPOP/test
> python test_21_performance.py TestPerformance.TestRandomMating

for both icc compiled and gcc compiled simuPOP (both for Python 2.6). Here is the result (execution time in seconds, the shorter the better):

It is still quite obvious that gcc is better in all cases, which comes as a surprise to me. Anyway, these numbers are from single runs and icc may not get the right optimization flags. If you have any suggestion, please feel free to let me know.

The comparison is done on a 3-year old DELL Precision 650 workstation with Dual Xeon CPU (3.73GHz) and 4G RAM, running RHEL5 x86-64. Just out of curiosity, I also run the same tests on a new PowerMac with a Quad-Core CPU (2.6G) and 8G of RAM, and a PC with a Quad-core CPU (Q6600) and 3G RAM, running 32bit of windows Vista. The benchmark from the Mac machine is impressive. It may be a good time to replace my linux workstation with a Mac. :-)