Creating a FLASH Site-Specific Makefile for the Dark Cluster

The Making of a Makefile

I recently identified a potential way to improve our process for running new simulations on FLASH on the Dark Cluster. While it isn't a high priority, I felt it would be a good exercise to familiarize myself with navigating using the terminal and customizing FLASH. I found it enlightening to document the issues I had and my eventual solution, hopefully for the benefit of others if this issue is encountered

Standard Procedure

When running a new simulation on the Dark Cluster using the latest version of FLASH, version 4.5, the process is fairly straightforward, but it most certainly could be optimized. After signing in, tools for compiling are loaded using the command module load intel/15.0.1 && module load intelmpi/5.0.2.044 . To save time on login, I have this command in my bash.rc file in my root directory, which runs on login. Once in the flash directory, one moves to flash/object and uses the command make clean to ensure that any files and links from the prior simulation are cleared. Moving back into flash , one initializes the desired simulation using the command ./setup [Simulation] -[Flags]. Some common flags used are -3d and -auto, which set the simulation to be run in 3 dimensions and automatically select the correct Makefile respectively. As an aside, the simulation being run is located in the directory flash/source/Simulation/SimulationMain with all of the necessary data to run the simulation located within a folder with the name of the simulation. I will likely explore the creation of a new complex simulation in a future blog post. 

This point is where I saw the potential for improving the efficiency of our process. Currently, we change into the object directory, delete the automatically generated Makefile.h and replace it with a preconfigured file that links all the correct dependencies and their directories. This is obviously not the most efficient means of accomplishing this, and there is potential for contributing to FLASH by adding the Dark Cluster to the list of sites with a preconfigured Makefile. I will elaborate on this further, but, for now, I will finish describing the process. 

After adding the correct Makefile.h,  one runs make in the flash/object directory, which compiles the simulation. However, I find it is much quicker to run this with the flag -j [threads], with [threads] being the number of threads the compilation is run on. I found 8 threads was more than sufficient to compile the simulation in a minute or so. 

At this point, there should be a file called flash.par located in flash/object. This is copied outside of the flash folder into a new folder. It's useful to title this the name of the simulation. After this, the bash file with the parameters for how the simulation should be run on the cluster (I call this jobScript.sh but the name isn't particularly important) is copied into the same folder. Changing directories to this folder, running the command sbatch jobScript.sh will start the simulation, and the files of the results should be placed in this folder once the simulation is done. 

A Note on Other Systems

Most of this procedure is the same on any Unix-based system, with some minor deviations. While the last paragraph is partially specific to the process used when working on the Dark Cluster, the basic process is likely similar on other systems, albeit with different syntax and a different way to initiate the simulation. The published documentation for a given system is usually enough to figure out how to run the simulation.

A Potential Improvement

When initializing FLASH, particularly when using the -auto flag, FLASH finds the hostname of the computer and compares it against a list of systems that have preconfigured  Makefiles contained within the FLASH package. This list can be found in the directory flash/sites. Also located here is the file Aliases , which contains a list of sites that report themselves with a hostname other than the one that can be used to connect to the site. For example, entering the hostname command while on the first Dark Cluster server returns the name fend01.cluster while the host name used to connect to the cluster is fend01.hpc.ku.dk

As of this writing, the Dark Cluster is not included in this list of sites. Because of this, when FLASH attempts to initialize the setup script, it attempts to identify and use the Makefile it believes is most suitable for the system it is running on. When running the command ./setup Sedov -3d -auto , or any ./setup file for that matter, FLASH uses the Makefile located in the directory flash/sites/fen.bluegenebnl.gov which does not specify the correct directories for FLASH dependencies. 

When attempting to continue the process of compiling a simulation using this Makefile, compilation in the flash/object directory with the make command fails because FLASH is unable to access the necessary MPI compilers that allow it to compile Fortran 95 code, which FLASH is largely written in. This is problematic because there is no clear indication that the Makefile is listing incorrect directories (in fact, FLASH actually prints SUCCESS upon initializing even if the Makefile is incorrect). 

Adding a new set of folders to the flash/sites directory for the Dark Cluster would be beneficial to reduce the potential points of failure in the process of compiling a simulation, but my attempts to do so have proved challenging.

Attempts at Execution

First off, there appears to be some disparity between the hostname reported when using the hostname command and the hostname used to connect to the Dark Cluster. The command returns fend01.cluster while the actual hostname used to connect is fend01.hpc.ku.dk . This is actually addressed by FLASH, which stores the file Aliases in the flash/sites directory. This file is used to make the hostname reported by the system an alias of the actual hostname. This allows FLASH to initialize using the correct preconfigured Makefile even if the hostname reported isn't the actual hostname. However, the Dark Cluster is not present in this file, and, as stated before, it is not one of the sites with a preconfigured Makefile.

When attempting to add a preconfigured Makefile, I ran into a few interesting issues that I hope can be enlightening. After adding a directory to flash/sites with the name fend01.hpc.ku.dk , I then added the Makefile provided by Aldo Batta in his extremely helpful guide on using FLASH on the Dark Cluster. Upon running ./setup Sedov -auto , FLASH defaulted to the fen.bluegenebnl.gov Makefile. Changing the directory name of the Dark Cluster Makefile to fend01.cluster yielded the same result, and adding the reported and correct hostnames to the Aliases file ended in the same outcome. Out of curiosity, I removed the line in Aliases that references fen.bluegenebnl.gov in an attempt to force FLASH to use the inserted Makefile. Upon doing this, FLASH used the inserted Dark Cluster Makefile and a basic Sedov simulation was able to run and I was able to verify that the data was useable in yt, the Python library we use for most analysis of hdf5 files. 
A basic slice plot of the temperature of the Sedov explosion towards the end of the simulation

I found it odd that FLASH was using the correct Makefile when the alias for fen.bluegenebnl.gov was removed. It was at this point that I noticed two things- I had placed my aliases for the Dark Cluster at the bottom of the list, and the alias for fen.bluegenebnl.gov was fen . Upon restoring the alias for fen.bluegenebnl.gov and moving the aliases I had set for the Dark Cluster above this alias, FLASH began to consistently use my inserted Makefile with the -auto flag. 

Conclusions

From what I can gather, FLASH searches the Aliases file for the first alias that contains part of the reported hostname. Because the alias for  fen.bluegenebnl.gov was fen, when FLASH searched for a match to fend01.cluster , it interpreted fen to be the correct alias because it is the first three letters of the reported hostname. upon moving the correct alias above this one, FLASH was able to match the reported alias correctly. While this may not be an issue that breaks the program, it certainly can be a minor inconvenience. Though I am currently not familiar enough with Fortran and the inner workings of the FLASH program to fix the underlying issue myself, this workaround is functional for my uses and likely will be for others.

 While this write up was certainly longer than it had to be to document this, I found it useful to commit to words the actual procedure of compiling the simulation and the process I went through while solving this issue. Admittedly, this post went into quite a bit of detail on these subjects, but hopefully it can be useful for others using FLASH and to look back on should trouble arise in the future.

Comments

Popular Posts