Softlink / Symbolic link in Windows NTFS

Softlinks are useful in any operating system to shorten long, complicated path names like C:/user/foo/data to just C:/data

Windows Command Prom

mklink LINK TARGET

For directories add the /d option

mklink /d LINK TARGET

Example

The default GNU Make filename in MinGW is mingw32-make.exe, but we’d rather type make:

cd C:\mingw\mingw64\bin

mklink make.exe mingw32-make.exe

Powershell

Powershell’s symlink creation syntax is more verbose:

New-Item -ItemType SymbolicLink -Path "Link" -Target "Target"

Example

cd C:\mingw\mingw64\bin

New-Item -ItemType SymbolicLink -Path "make.exe " -Target "mingw32-make.exe"

Notes

ln -s /media/myusbdrive/data ~/data

Notice that the order of target arguments are reversed between Windows and Linux.

Fix Spyder IDE not visible

Spyder IDE is a complex but usually stable Python program. When something goes wrong with Spyder, often the symptom is it simply won’t appear, maybe you’ll just get the splash logo. To totally reset Spyder (erasing all user preferences for Spyder), type in Terminal / Command Prompt:

spyder --reset

Normally, that fixes Spyder.

To diagnose further, start Spyder from Terminal instead of OS Start menu, it might give some hints.

CUDA, cuDNN and NCCL for Anaconda Python

In 2017, Anaconda Accelerate was discontinued. The GPU CUDA, cuDNN and NCCL functionality are accessed in a Numpy-like way from CuPy. CuPy also allows use of the GPU is a more low-level fashion as well.

Before starting GPU work in any programming language realize these general caveats:

  • I/O heavy workloads may make realizing GPU benefits more difficult
  • Consumer GPUs (GeForce) can be > 10x slower than workstation class (Tesla, Quadro)

Prereqs

You must have a discrete Nvidia GPU in your laptop or desktop. Check for existence of an Nvidia GPU in your computer by:

Linux

lspci | grep -i nvidia

a blank response means an Nvidia GPU is not detected.

Windows

dxdiag

Look under the “render” tab to see if an Nvidia GPU exists.

Install

  1. Determine the Compute Capability of your model GPU and install the correct CUDA Toolkit version.
  2. CuPy is installed distinctly depending on the CUDA Toolkit version installed on your computer.
  3. reboot or import cupy will fail with errors like:

AttributeError: type object ‘cupy.core.core.Indexer’ has no attribute ‘reduce_cython

Check CuPy

CuPy syntax is very similar to Numpy. There are a large set of CuPy functions relevant to many engineering and scientific computing tasks.

import cupy

dev = cupy.cuda.Device()
print('Compute Capability', dev.compute_capability)
print('GPU Memory', dev.mem_info)

The should return like:

Compute Capability 75

If you get error like

cupy.cuda.runtime.CUDARuntimeError: cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime version

This means the CUDA Toolkit version is expecting a newer Nvidia driver. The Nvidia driver can be updated via your standard Nvidia update program that was installed from the factory. “Table 1” of the CUDA Toolkit release notes gives the CUDA Toolkit required Driver Versions.

Examples

Notes

Alternatives to CuPy include Numba.cuda, which is a lower-level C-like CUDA interface from Python.

CUDA for Julia is provided in JuliaGPU.

Code cells in Python IDE

A code cell in popular Python IDEs including PyCharm and Spyder is created by line starting with # %%. This “code cell” is analogous to IPython code cells and Matlab code sections.

You will see like

import math

# %% user data
x = 3
y = 4
# %% main loop
for i in range(5):
    x += y

The code cells allow running sections of code in an IDE without the need to constantly set/unset breakpoints in the IDE. They also catch the eye of developers to delineate logical blocks of code in the algorithm.

We encourage the use of code cell syntax, even if you don’t use them in the IDE directly, as the IDE will highlight sections of code to visibly delineate these separate parts of the algorithm.

Install PGI free Fortran compiler

Related: Install Flang LLVM Fortran compiler


The PGI Community Edition compilers are a set of no-cost, non-open-source, high performance compilers for:

  • C pgcc
  • C++ pgc++
  • Fortran pgfortran including CUDA Fortran support.

Like Flang, PGI supports significant portions of modern Fortran 2008 / 2018 syntax.

Install

Download and install PGI Community Edition:

  • Linux / Mac: ~/.local/pgi sudo not required, but must be on symbolic-link-aware drive (not ExFAT)
  • Windows: d:\PGI (can be on any drive, including ExFAT). Requires Visual Studio as well.

This is over 2 GB download. You don’t need to install CUDA support unless you have and want to use a CUDA-capable GPU.

Linux / Mac

add to ~/.bashrc:

export PATH="$PATH:$HOME/.local/pgi/linux86-64/2019/bin/"

or whichever directory you find with

find $HOME/.local/pgi -name pgfortran

Open a new terminal to use PGI pgfortran pgcc pgc++

Windows

add to PATH (ensure pgfortran.exe is there):

d:\PGI\win64\2019\bin

Usage

Sadly, PGI 19.7 deprecated PGI debugger. PGI Java-based pgdbg graphical debugger was for Fortran, C and C++ and was in the no-cost Community Edition as well. The program to be debugged needs compile options -g -O0 to provide maximum debugging information.

Example with hello.f90:

  1. pgfortran -g hello.f90
  2. pgdbg a.out opened the graphical Fortran debugger.

If you don’t see code in the upper left of the graphical PGI debugger for your program, be sure you compiled the executable with -g -O0 options.

CMake

CMake ≥ 3.10 distinguishes between PGI and Flang compilers. Set compiler-specific options in CMakeLists.txt for various Fortran compilers like:

cmake_minimum_required (VERSION 3.10)
project(myproj Fortran)

if(CMAKE_Fortran_COMPILER_ID STREQUAL Intel)
  if(WIN32)
    add_compile_options(/stand:f18 /warn:declarations)
  else()
    add_compile_options(-stand f18 -warn declarations)
  endif()
elseif(CMAKE_Fortran_COMPILER_ID STREQUAL GNU) # gfortran
  add_compile_options(-fstd=f2018 -fimplicit-none)
elseif(CMAKE_Fortran_COMPILER_ID STREQUAL PGI)
  add_compile_options(-Mdclchk)
elseif(CMAKE_Fortran_COMPILER_ID STREQUAL Flang) # cmake >= 3.10

endif()

and then clear out your build directory and select the PGI compilers with

FC=pgfortran CC=pgcc CXX=pgc++ cmake ..

Meson

Meson ≥ 0.50 supports PGI and Flang compilers.

project('demo','fortran')

fc = meson.get_compiler('fortran')
if fc.get_id() == 'pgi'
  message('PGI compiler')
elif fc.get_id() == 'flang'
  message('Flang compiler')
endif

Aspell for Windows

The native Windows Aspell program was quite broken last time I tried it. Instead, use Aspell in Windows Command Prompt via Windows Subsystem for Linux.

Setup

From WSL Terminal:

apt install aspell

In Windows, create a file say in %userprofile%/scripts/aspell.bat and add %userprofile%/scripts to your Windows PATH.

file aspell.bat contains:

@echo off

bash -c "aspell check %1"

Usage

From Windows Command Prompt:

aspell myfile.txt

or

aspell path/to/myfile.txt

don’t use backslash \ in paths, only forward slash /

Limitations of loading HDF5 files with xarray

xarray.open_dataset gained the ability to open HDF5 files in 0.12.0. However, this can cause Python to quietly crash without error message, which can be quite confusing. This is true even with the minimum required versions of xarray, h5py and h5netcdf installed.

We don’t have a specific workaround for this other than to use h5py directly. We have used h5py for several years in high-stakes operations, including data analysis and data collection. Both h5py and netcdf4 Python modules work with context managers to avoid excess I/O resource consumption.

Using Git SSH with GitLab self-managed instances

Since GitLab Community Edition is open source, large projects like CMake may host their own self-managed GitLab instance. To make merge requests to such projects, one can configure Git SSH. For this example, we use Kitware’s CMake GitLab instance https://gitlab.kitware.com/cmake. This procedure works for any operating system, including Windows.

First, create an account on the self-managed GitLab instance and fork the desired repo. This will be available like

git clone https://gitlab.kitware.com/username/cmake

To git push using SSH, type:

git config --global url."ssh://gitlab.kitware.com/".pushInsteadOf https://gitlab.kitware.com/

Generate an SSH key–don’t reuse SSH keys between sites.

ssh-keygen -t ed25519 -f ~/.ssh/kitware

Go to the GitLab SSH Key page like

https://gitlab.kitware.com/profile/keys

and add the contents of ~/.ssh/kitware.pub

Add to ~/.ssh/config:

Host gitlab.kitware.com
  User git
  IdentityFile ~/.ssh/kitware

Now checkout a new branch, make your changes according to Contributing.rst and submit a merge request.

Duplicate GitHub Wiki

Related: Moving GitHub Wiki


The GitHub API v4 does not include Wikis. The GitLab API v4 does include Wikis.

Thus, to duplicate a GitHub repo AND the GitHub Wiki benefits from a little scripting help.

An example solution is in GitEDU. This requires manually clicking to enable the new wiki via a web browser for each repo wiki.

GitHub Wiki Git accessibility

In general, GitHub Wiki is just another Git repo. The URL for the GitHub Wiki is obtained by appending .wiki.git to the associated GitHub repo URL.

Example:

  • Main repo is https://github.com/username/reponame
  • Wiki repo is https://github.com/username/reponame.wiki.git

Moving a GitHub Wiki

Related: Duplicating GitHub Wiki


GitHub Wikis are not accessible via GitHub API v4. We can use a couple simple Git command line statements to move a GitHub Wiki.

For these examples, we assume that:

  • old wiki https://github.com/username/repo1.wiki.git
  • new wiki https://github.com/username/repo2.wiki.git

Copy GitHub Wiki to your laptop:

git clone --bare https://github.com/username/repo1.wiki.git

Browse to new Wiki https://github.com/username/repo2/wiki and create blank Wiki

Mirror push Wiki to new repo:

git -C repo1 push --mirror https://github.com/username/repo2.wiki.git

Once you see the new Wiki https://github.com/username/repo2/wiki is OK, remove the old Wiki pages if desired:

git -C repo1 rm *.md

git -C repo1 commit -am "deleted old wiki pages"

git -C repo1 push