Get Matlab HDF5 version

Matlab upgraded to HDF5 1.8.12 in R2015a. Matlab is still at HDF5 1.8.12 at least through R2019b. This means to ensure interoperability with files from h5py and other such HDF5-using software, one should currently use HDF5 1.8 files. Newer versions of the HDF5 library (e.g. 1.10) can write HDF5 1.8 files. HDF5 1.8.12 was released in November 2013. HDF5 1.8.21 was released in June 2018 and is perhaps the last HDF5 1.8 release.

Check Matlab HDF5 version

Matlab R2019b shows HDF5 1.8.12 is used:

[majnum,minnum,relnum] = H5.get_libversion()

Ninja bootstrap build

Ninja v1.10.0 fixes the longstanding issues with CMake + Fortran that led Kitware to maintain its own Ninja fork. Ninja uses GitHub Actions to build binaries, but a recent glitch caused Ninja v1.10.0 source to be released without binaries. Ninja is quick to compile by:

python configure.py --bootstrap

On Windows, Ninja requires building from a Visual Studio prompt, perhaps x64 Native.

Ninja build on CentOS 7

The binary executables for Ninja 1.9.0 do not work on CentOS 7, the error is like:

$ ninja

ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ninja)
ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ninja)

This was a known issue with the Ninja release artifact build process that was fixed in Ninja 1.10.0.

Workaround

If Ninja 1.10.0 is not available, use Ninja 1.8.2 on CentOS. This may work for other “older” Linux distros with similar errors.

Using Intel compilers and MKL with CMake and Make

There can be substantial speed boosts from using Intel compilers. Intel Parallel Studio gives advanced debuggers and performance measurements. Intel MKL can give a significant speed boost even to non-Intel compilers for certain math operations.

Linux / MacOS

To build a CMake project:

FC=ifort CC=icc CXX=icpc cmake -B build

cmake --build build

Windows

To select a non-Visual Studio backend on Windows, configure like:

cmake -G Ninja

or

cmake -G "MinGW Makefiles"

a complete example for Windows:

set FC=ifort
set CC=icl
set CXX=icl

cmake -G Ninja -B build

cmake --build build

Intel MKL with CMake

MKL can be used with any compiler, e.g. ifort or gfortran. An example CMakeLists.txt using the factory FindLAPACK.cmake:

project(MKLtest Fortran)

# allows selecting parallel, sequential, 32/64 bit
# This example is sequential 32 bit

set(BLA_VENDOR Intel10_64lp_seq)
find_package(LAPACK REQUIRED)

add_executable(mytest main.f90)
target_link_libraries(mytest ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES})

verbose build output

In general to see the compiler commands CMake is issuing, use

cmake --build build -v

Refer to Intel Link Advisor.

Verify MKL is used

The CMake script falls back to non-MKL options if MKL is not installed. Get runtime confirmation that MKL is being used via MKL_VERBOSE.

  • Linux / MacOS:

    MKL_VERBOSE=1 ./mytest
    
  • Windows

    set MKL_VERBOSE=1
    mytest.exe
    

That gives verbose text output upon use of MKL functions. That runtime option does slow down MKL performance, so normally we don’t use it.

Notes

An easier to use FindLAPACK.cmake

print compiler macro definitions

Compilers define macros that can be used to identify a compiler and platform from within C, C++ and Fortran code. This can be useful for many purposes where short bits of platform-specific or compiler-specific code is needed. If a significant amount of code is needed, it may be better to swap in different code files using the build system instead of lengthly #if defined(foo) logic. There are numerous examples for C and C++ so here we will focus on macros of Fortran compilers.

Gfortran

Gfortran compiler macro definitions are obtained in an OS-agnostic way by:

echo "" | gfortran -dM -E - > macros.txt

that creates a file “macros.txt” containing all the compiler macros.

commonly used macros to detect operating system / compiler configuration include:

  • _WIN32 1
  • __linux__ 1
  • __unix__ 1
  • __APPLE__ 1

Intel Fortran

Intel Fortran compiler macros include the Gfortran macros noted above and additionally:

  • __INTEL_COMPILER 1

PGI Fortran

PGI Fortran compiler macros are printed by:

pgfortran -dM

the PGI macros include the Gfortran macros above as well as:

  • __PGI 1

Flang

Flang macros include

  • __FLANG 1

Other compilers

Other Fortran compiler macros that identify the compiler and platform can be found in CMake source code.

CMake FetchContent vs. ExternalProject

Making multiple software projects work together is usually better done by the build system:

instead of Git submodule.

Meson subproject and CMake ExternalProject keep project namespaces separate. Meson subproject and CMake FetchContent download and configure all projects at configure time. CMake FetchContent comingles the CMake project namespaces. FetchContent can be easier to use than ExternalProject if you control both software projects’ CMake scripts. If you don’t control the “child” project, it may be better to use ExternalProject instead of FetchContent.

For these examples, suppose we have a top-level project “parent” and a “child” project containing a library that is desired in parent. Suppose the child project can be built standalone (by itself) but also may be used directly from other CMake projects.

projectCMAKE_SOURCE_DIRCMAKE_BINARY_DIRPROJECT_SOURCE_DIR
parent~/foo~/foo/build~/foo
child: standalone~/bar~/bar/build~/bar
child: CMake ExternalProject~/foo/build/child-prefix/src/child~/foo/build/child-prefix/src/child-build~/foo/build/child-prefix/src/child
child: CMake FetchContent~/foo~/foo/build~/foo/build/_deps/child-src

FetchContent

FetchContent populates content from the other project at configure time. FetchContent populates the “child” project with default values from the “parent” project. Varibles set in the “child” project generally do not affect the “parent” project unless specifically used from the “parent” project.

From “parent” project CMakeLists.txt:

cmake_minimum_required(VERSION 3.14)
project(parent Fortran)

include(FetchContent)
FetchContent_Declare(child
  GIT_REPOSITORY https://github.invalid/username/child.git
  GIT_TAG master   # it's much better to use a specific Git revision or Git tag for reproducibility
)

FetchContent_MakeAvailable(child)

# your program
add_executable(myprog main.f90)
target_link_libraries(myprog mylib)  # mylib is from "child"
FetchContent_MakeAvailable
make “child” code configure, populating variables and targets as if it were part of “parent” CMake project.

suppose “child” project CMakeLists.txt contains:

project(child Fortran)

add_library(mylib mylib.f90)
target_include_libraries(mylib INTERFACE ${CMAKE_CURRENT_BINARY_DIR}/include)
set_target_properties(mylib PROPERTIES
  Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/include)

The child project CMAKE_BINARY_DIR and CMAKE_SOURCE_DIR will be those of parent project. That is, if the parent project is in ~/foo and the build directory is ~/foo/build, then the child project in ~/childcode called by FetchContent will also have CMAKE_SOURCE_DIR of ~/foo and CMAKE_BINARY_DIR of ~/foo/build. So be careful in the child project when using such variables that may be defined by parent projects. This is why projects that aren’t specifically designed to work together may be better joined by ExternalProject. A typical technique within the child project that can operate standalone is to refer to CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_SOURCE_DIR as the latter will break when used from FetchContent.

ExternalProject

ExternalProject populates content from the other project at build time. This means the other project’s libraries are not visible until the parent project is built. Since ExternalProject does not combine the project namespaces, ExternalProject may be necessary if you don’t control the other projects.

ExternalProject will not download, configure or build without the add_dependencies() statement. Upon cmake --build of the parent project, ExternalProject downloads, configures and builds.

From “parent” project CMakeLists.txt:

project(parent Fortran)

include(ExternalProject)

ExternalProject_Add(child_proj
  GIT_REPOSITORY https://github.com/scivision/cmake-externalproject
  GIT_TAG master  # it's much better to use a specific Git revision or Git tag for reproducability
  INSTALL_COMMAND ""  # this disables the install step for the external project
)

ExternalProject_Get_Property(child_proj BINARY_DIR)

file(MAKE_DIRECTORY ${BINARY_DIR}/include)  # avoid race condition

add_library(timestwo STATIC IMPORTED GLOBAL)
set_target_properties(timestwo PROPERTIES
  IMPORTED_LOCATION ${BINARY_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}timestwo${CMAKE_STATIC_LIBRARY_SUFFIX}
  INTERFACE_INCLUDE_DIRECTORIES ${BINARY_DIR}/include)

add_executable(test_timestwo test_timestwo.f90)  # your program
add_dependencies(test_timestwo child_proj)  # externalproject won't download without this
target_link_libraries(test_timestwo timestwo)
add_dependencies()
make ExternalProject always update and build first

The imported library ext is used in the “parent” project just like any other library.


“child” project CMakeLists.txt includes:

project(child Fortran)

add_library(timestwo STATIC timestwo.f90)
set_target_properties(timestwo PROPERTIES
  Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/include)

Configure “child” Fortran_MODULE_DIRECTORY so that it’s not necessary for “parent” to introspect “child” directory structure.

Live examples

Caveats

Since the ExternalProject is built by itself and generally is unaware of the consuming “parent”, this does NOT work to detect use as an ExternalProject:

project(child ...)

# "is_fetched" is:
# * ExternalProject: false--does not detect
# * FetchContent: true

set(is_fetched (NOT CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR))

Note that the PARENT_DIRECTORY property is NOT useful for detecting if the “child” is being used as an ExternalProject.


  • target_link_directories() is generally NOT preferred because library name collisions can occur, particularly with system libraries.

Installing LLVM Flang Fortran compiler

Since 2015, NVIDIA has supported Flang: a Fortran compiler frontend to LLVM. Flang f18 targets modern Fortran 2008 / 2018 syntax and is implemented in C++17 internally. Flang may become part of LLVM 11, as per the LLVM-Dev mailing list. The binary download release of Flang hasn’t been updated since early 2019. To get the current version of Flang, you may need to build yourself.

Build system support

Flang is supported by Cmake ≥ 3.10, although CMake ≥ 3.14 is recommended in general to support more Fortran 2008 features.

Meson ≥ 0.50 supports Flang and in general Meson is easier to use and more powerful than CMake.

Select Flang in the build system as usual by setting environment variables:

FC=flang

How to install Flang

Pick ONE of the following:

Download Flang binary

Flang prerelease binaries are provided for Linux.

Download Flang binary and extract to ~/.local/flang

Add to ~/.bashrc

export PATH=$PATH:$HOME/.local/flang/bin
export LD_LIBRARY_PATH=$HOME/.local/flang/lib/:$LD_LIBRARY_PATH

Open a new Terminal and check

flang --version

Building Flang

Building the LLVM Flang Fortran compiler from source is a resource-intensive process.

  • A moderately powerful PC is needed to compile Flang with > 100 GB of free space on the drive you build Flang on. Flang install is small, but Flang build process requires a lot of drive space.
  • Don’t use too much RAM with the -j option of make for building Flang. It maxed out RAM on a PC with 32 GB of RAM with make -j. Try just plain make.
  • follow the Flang build instructions sequentially. Don’t try to build later parts while earlier parts are still building, this will not work properly.

Notes

  • Flang missed LLVM 10 merge window

List all CMake tests with CTest

As a CMake project grows, the increasing complexity can make it hard to remember what tests are to be run. Perhaps the project logic is unexpectedly omitting necessary tests. The CI system or human can verify the list of tests is as expected by parsing the simple text output from:

ctest -N

before this, the project must be configured and built as usual:

cmake -B build
cmake --build build --parallel
cd build
ctest -N

Git commit date / time / author edit

If the Git commits have already been push to remote, this process will require other users of the repo to reset or reclone. That’s true with any Git operation that edits history. If the Git commits have not already been pushed, then this process will not require extra steps from other repo users.

show commit AuthorDate CommitDate

In general, show any commit’s AuthorDate and CommitDate by

git show <commit_hash> --pretty=fuller

for the most recent commit, simply:

git show --pretty=fuller

edit last commit only

To reset the author of the last commit to the current Git username and email, as well as setting AuthorDate and CommitDate to the current time:

git commit --amend --reset-author --no-edit
--reset-author
reset date/time/author to current
--no-edit
skip opening text editor

edit previous commits, including already pushed

Use git rebase -i as usual and for the commits to reset author / date, change the operation to e to edit each by:

git commit --amend --reset-author --no-edit

Reference

GitHub commit troubleshooting

HDF5 on Intel Fortran for Windows

Intel Fortran on Windows provides an easy way to use Fortran MPI on Windows. The Intel Fortran compile and link commands on Windows are distinct from those on Linux / MacOS, perhaps reflecting the internal use of Visual Studio on Windows. The HDF5 1.10.6 release changed the naming convention for the HDF5 Fortran library files on all operating systems.

  • old: hdf5hl_fortran.
  • new: hdf5_hl_fortran.

CMake’s FindHDF5.cmake did not have this change in CMake 3.16.2. We created a CMake Issue for this.

Tentatively one can compile using HDF5 with Intel Fortran on Windows like:

ifort -I"C:/Program Files/HDF_Group/HDF5/1.10.6/include/shared" -I"C:/Program Files/HDF_Group/HDF5/1.10.6/include/static" test_minimal.f90 "C:/Program Files/HDF_Group/HDF5/1.10.6/lib/hdf5_fortran.lib" "C:/Program Files/HDF_Group/HDF5/1.10.6/lib/hdf5_hl_fortran.lib" "C:/Program Files/HDF_Group/HDF5/1.10.6/lib/hdf5.lib" "C:/Program Files/HDF_Group/HDF5/1.10.6/lib/szip.lib" "C:/Program Files/HDF_Group/HDF5/1.10.6/lib/zlib.lib"

If you use include/static you will get errors like

error LNK2019: unresolved external symbol H5GLOBAL_mp_H5F_ACC_TRUNC_F referenced in function MAIN__