There may come a time when you’ll find it useful to utilize a library for one of your projects. Building a library from source code is a fairly common practice and consists of the same general steps. Recently we received a query regarding how to install Armadillo, a matrix math library for C++. We can use the installation of Armadillo as an example for seeing the process in detail. We’ll then build a test app that uses functions defined in the library.
The process follows these four general steps:
- Download source code and determine build system
- Configure (find dependent software, enable desired features)
- Build and install
- Compile test code for verification
Download source code
First, let’s create a directory specifically for this install. We recommend installing software to a directory such as /scratch/<username>/software/armadillo where <username> is your USC username. This directory will be referred to as $ARMADILLO_ROOT.
mkdir -p $ARMADILLO_ROOT/src # create home for source code
cd $ARMADILLO_ROOT/src
Armadillo source code is available here: Armadillo: C++ library for linear algebra & scientific computing - Download We can download it with the wget command in the terminal.
wget --no-check-certificate http://sourceforge.net/projects/arma/files/armadillo-10.8.2.tar.xz
You should see a file named armadillo-10.8.2.tar.xz, which can then be extracted with tar like so:
tar xf armadillo-10.8.2.tar.xz
The option x will extract, while f specifies which file to expand. Sometimes you may see a v option for verbose that will display the latest extracted file as a sort of progress report.
There should now be a directory named armadillo-10.8.2 with the source code and build scripts inside.
$ ls -F armadillo-10.8.2
armadillo_icon.png configure* NOTICE.txt
armadillo_joss_2016.pdf docs.html rcpp_armadillo_csda_2014.pdf
armadillo_lncs_2018.pdf examples/ README.md
armadillo_nicta_2010.pdf include/ src/
armadillo_solver_2020.pdf index.html tests1/
armadillo_spcs_2017.pdf LICENSE.txt tests2/
cmake_aux/ mex_interface/
CMakeLists.txt misc/
There are two options for installation. The first is to run a configure script and the second is to use cmake. Cmake is preferred because it does a good job of tracking dependent software.
If you’re curious, you can look at cmakelists.txt, which contains the build instructions. As a user of the software, you normally shouldn’t have to make changes here or even really look at the contents.
Create a new directory to start the build, traditionally just named build. If something goes wrong, we can always delete this directory and start again. If you don’t complete this step, the installation process creates intermediate build files in the working directory (sometimes referred to as an in-source build) that may be hard to clean up if something goes wrong.
mkdir build
cd build
module load cmake
Configuration
cmake normally requires several configurations to get everything working properly. Typically, you can read the documentation for a description of options you will need. For now, let’s start with the basics and add options as needed.
The Armadillo documentation recommends these optional dependencies:
- OpenBLAS
- LAPACK
- ARPACK
- SuperLU
CARC has OpenBLAS (which provides LAPACK) and SuperLU available as modules. ARPACK is not available, so for simplicity we will not include it.
Start the cmake configuration using this command:
cmake \
-DCMAKE_INSTALL_PREFIX=$ARMADILLO_ROOT \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
..
This command may look a bit unfamiliar, so let’s break it down before we continue.
The -D options override default cmake settings. CMAKE_INSTALL_PREFIX specifies where Armadillo gets written to when complete (usually /usr/ by default). The CMAKE_C{XX}_COMPILER options specify which C and C++ compilers to use.
The \ characters at the end of each line are line breaks and help make the command more readable. Finally, the .. line is the path to the CMakeLists.txt file (the parent directory).
cmake will then scan for dependencies and print out test results for what it finds. It will print a lot of information but here are a few things of note:
cmake finds OpenBLAS but not LAPACK.
-- MKL_FOUND = NO
-- OpenBLAS_FOUND = YES
-- ATLAS_FOUND = NO
-- BLAS_FOUND = NO
-- LAPACK_FOUND = NO
--
-- *** NOTE: if OpenBLAS is known to provide LAPACK functions, recommend to
-- *** NOTE: rerun cmake with the OPENBLAS_PROVIDES_LAPACK option enabled:
-- *** NOTE: cmake -D OPENBLAS_PROVIDES_LAPACK=true .
cmake is unable to find hdf5, ARPACK, and SuperLU.
--
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Checking for module 'hdf5'
-- No package 'hdf5' found
-- HDF5_FOUND =
-- ARPACK_FOUND = NO
-- Looking for SuperLU version 5
-- Could not find SuperLU
-- SuperLU_FOUND = NO
A Makefile has been created and Armadillo will compile now but optional dependencies were not found.
$ ls
ArmadilloConfig.cmake CMakeFiles Makefile
ArmadilloConfigVersion.cmake cmake_install.cmake tmp
CMakeCache.txt InstallFiles
If we build as is, some features will not work properly or will run slower. In some cases, if dependent software is not found by cmake, the build will not be able to proceed.
The dependencies we are are going to use are available as modules, but are built with gcc 11.2.0. Let’s set our environment by changing compilers. We will also have to clean out all the files generated from our last attempt to make sure there’s no “contamination” from our previous environment.
Delete old files:
# Only run this rm command in the build directory!
# It will delete EVERYTHING in the current directory
rm -r *
Set our environment:
$ module purge
$ module load gcc/11.2.0 openblas openmpi hdf5 superlu cmake
$ module list
Currently Loaded Modules:
1) gcc/11.2.0 3) openmpi/4.1.2 5) superlu/5.3.0
2) openblas/0.3.19 4) hdf5/1.12.1
Now let’s rerun cmake in the newly cleaned directory.
cmake \
-DCMAKE_INSTALL_PREFIX=$ARMADILLO_ROOT \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DOPENBLAS_PROVIDES_LAPACK=true \
..
cmake outputs some new messages, but most of it we’ve seen before. This is an indication that we’re heading in the right direction:
-- *** Result of configuration:
-- *** ARMA_USE_WRAPPER = true
-- *** ARMA_USE_LAPACK = true
-- *** ARMA_USE_BLAS = true
-- *** ARMA_USE_ATLAS = false
-- *** ARMA_USE_HDF5_ALT = true
-- *** ARMA_USE_ARPACK = false
-- *** ARMA_USE_EXTERN_RNG = true
-- *** ARMA_USE_SUPERLU = true
We should now see a Makefile in the current directory. We can run make to build the library. You should see something like this:
$ make
[ 33%] Building CXX object CMakeFiles/armadillo.dir/src/wrapper1.cpp.o
[ 66%] Building CXX object CMakeFiles/armadillo.dir/src/wrapper2.cpp.o
[100%] Linking CXX shared library libarmadillo.so
[100%] Built target armadillo
If something goes wrong, the problem most likely occurred at the cmake stage. Determining what needs to be changed in the cmake command is probably the most time-consuming part of the process. Once you figure out how to get it built properly, it may be useful to keep a record of both the cmake options you used along with the modules currently loaded by running module list.
$ module list
Currently Loaded Modules:
1) gcc/11.2.0 3) openmpi/4.1.2 5) superlu/5.3.0
2) openblas/0.3.19 4) hdf5/1.12.1 6) cmake/3.21.3
Installing
Assuming everything went smoothly, we can now install with make install. This command will copy over files to the install prefix we specified earlier ($ARMADILLO_ROOT). Traditionally, these files include library files (.so or .a files), headers (.h files), and cmake configuration files for future projects that may use both cmake and Armadillo.
Here’s a snippet of what you should see:
make install
[100%] Built target armadillo
Install the project...
-- Install configuration: ""
-- Up-to-date: /scratch1/ttroj/playground/software/armadillo/include
-- Up-to-date: /scratch1/ttroj/playground/software/armadillo/include/armadillo_bits
-- Up-to-date: /scratch1/ttroj/playground/software/armadillo/include/armadillo_bits/fn_conv_to.hpp
-- Up-to-date: /scratch1/ttroj/playground/software/armadillo/include/armadillo_bits/spglue_times_meat.hpp
-- Up-to-date: /scratch1/ttroj/playground/software/armadillo/include/armadillo_bits/glue_cross_meat.hpp
.
.
.
Build test app
To test that everything is working, Armadillo provides source code for an example program in $ARMADILLO_ROOT/src/armadillo-10.8.2/examples.
Let’s try to build it. Before we start, make sure you have the following modules loaded:
$ module list
Currently Loaded Modules:
1) gcc/11.2.0 3) openmpi/4.1.2 5) superlu/5.3.0
2) openblas/0.3.19 4) hdf5/1.12.1
From the Armadillo documentation we should be able to run:
g++ example1.cpp -o prog -std=c++11 -O2 -larmadillo
We get this message:
example1.cpp:2:10: fatal error: armadillo: No such file or directory
#include <armadillo>
^~~~~~~~~~~
compilation terminated.
This error means that the preprocessor would like to include a file named armadillo to example1.cpp, but is unable to find it.
Since Armadillo is installed in a “non-standard” location (somewhere other than a global location like /usr/), we need to tell g++ where to find the header and include files with the -I option. Remember, we set the environment variable $ARMADILLO_ROOT to the directory in which we installed Armadillo.
The updated command is then:
$ g++ -I$ARMADILLO_ROOT/include example1.cpp -o prog -std=c++11 -O2 -larmadillo
/usr/bin/ld: cannot find -larmadillo
collect2: error: ld returned 1 exit status
Now g++ needs to know where the Armadillo library files are saved. We can add the -L option like so:
g++ -I$ARMADILLO_ROOT/include example1.cpp -o prog -std=c++11 -O2 -L$ARMADILLO_ROOT/lib64 -larmadillo
If successful, you should now see an executable named prog. If you see an error message similar to:
/usr/bin/ld: warning: libhdf5.so.200, needed by /scratch1/ttroj/playground/software/armadillo/lib64/libarmadillo.so, not found (try using -rpath or -rpath-link)
/scratch1/ttroj/playground/software/armadillo/lib64/libarmadillo.so: undefined reference to `H5Dread'
/scratch1/ttroj/playground/software/armadillo/lib64/libarmadillo.so: undefined reference to `H5Sget_simple_extent_ndims'
/scratch1/ttroj/playground/software/armadillo/lib64/libarmadillo.so: undefined reference to `H5T_NATIVE_SCHAR_g'
/scratch1/ttroj/playground/software/armadillo/lib64/libarmadillo.so: undefined reference to `H5Eset_auto2'
.
.
.
Double check that you have the correct modules loaded.
Let’s try to run it now:
./prog
./prog: error while loading shared libraries: libarmadillo.so.10: cannot open shared object file: No such file or directory
Kind of frustrating, right? What’s happening now is that the binary can’t find the Armadillo library. You can run ldd prog to see what libraries are required and where they are found. This command will output a lot of text to search through, so we can pipe the output into grep to see what’s missing:
$ ldd prog | grep "not found"
libarmadillo.so.10 => not found
By default, libraries can be found if they are installed in global locations such as /usr/lib. Additionally, directories listed in $LD_LIBRARY_PATH will be searched. In fact, these directories take priority over ones in the default paths.
You may notice a few libraries within /spack/apps2/linux-centos7-x86_64/gcc-11.2.0. It’s possible to embed paths in the binary during the build process. This method is preferred since you don’t need to set environment variables that you might forget about later—which is especially useful if you don’t want to go through the effort of building a module for your libraries.
We’ll have to rebuild and use the rpath option. The syntax looks like -Wl,-rpath,/path/to/lib.so
g++ -I$ARMADILLO_ROOT/include example1.cpp -o prog -std=c++11 -O2 -L$ARMADILLO_ROOT/lib64 -Wl,-rpath,$ARMADILLO_ROOT/lib64 -larmadillo
You should now be able to run prog, even after doing a module purge:
$ ./prog
Armadillo version: 10.8.2 (Realm Raider)
A.n_rows: 2
A.n_cols: 3
A:
0 0 0
0 0 4.5600e+02
A:
5.0000
A:
5.0000 5.0000 5.0000 5.0000 5.0000
5.0000 5.0000 5.0000 5.0000 5.0000
5.0000 5.0000 5.0000 5.0000 5.0000
5.0000 5.0000 5.0000 5.0000 5.0000
A:
0.1653 0.4540 0.9958 0.1241 0.0471
0.6888 0.0365 0.5528 0.9377 0.8664
0.3487 0.4794 0.5062 0.1457 0.4915
0.1487 0.6823 0.5712 0.8747 0.4446
0.2457 0.5952 0.4093 0.3678 0.3857
...
...
You’re all set to start using Armadillo! You can apply these same steps to build whatever libraries you need for your current projects.