Create HDF5 soft links with HDFView

HDF5 soft (symbolic) links work like in filesystems–convenient as HDF5 UI is like a filesystem in a file. An unlimited number of soft links can point to the same resource. Soft links can dangle (point to an missing location) if needed.

HDFView

The HDFView GUI can create softlinks as well. This is handy when the program code changes variable name and one wants to keep HDF5 files compatible with old and new code (old name and new name). To create an HDF5 soft link in HDFView:

  1. right click target variable name
  2. select New → Link → Soft Link
  3. when done, use the “close” button to save these changes

Note: if HDFView User Options are set to “Read Only” the “New” right-click option will be grayed out.

The soft link appears as any other variable in the HDFView GUI.

HDFView and other HDF5 interfaces can also create hard links and external links. An unlimited number of hard links can point to the same resource like soft links, but hard links are not allowed to dangle.

External links create a soft link to a resource in another HDF5 file. External links can be useful where datasets are very large, but one can treat a collection of files as one gigantic file from the user perspective.

References

CMake file separator

In many cases, using the Unix-type slash file separator / will work even on Windows. Trying to manually specify backslash Windows file separator \ can cause problems in CMake and many other languages. Thus in CMake and other languages like Python, we always use / as path separator, and only transform to backslash for the rare cases it’s needed.

Transform to Unix file separator

file(TO_CMAKE_PATH …) switches backslash \ file separators to Unix slash / file separators. This becomes relevant if manually adjusting Include paths by appending to _INCLUDE_DIRS or similar. If backslashes sneak through, all kinds of weird build-time errors can result, and even configure-time errors if you use “check_c_source_compiles()” and similar. As the docs note, put quotes "${mypath}" around the variable expansion to ensure CMake doesn’t mangle the path.

The source code is in cmFileCommand.cxx in function HandlePathCommand. Many internal factory CMake scripts use TO_CMAKE_PATH without if() guards.

Example

This CMake script can be used like

cmake -Din=c:\foo\bar -P to_cmake_path.cmake
file(TO_CMAKE_PATH "${in}" out)

message("${in} => ${out}")

c:/Program Files/CMake

Transform to native file separator

This would be much more rarely used, but if needed, CMake can transform paths to native file separator, with the caveat that this can cause unpredictable Windows-specific backslash problems, as with any program.

file(TO_NATIVE_PATH ...)

Matlab geoglobe Earth visualizations

The Mathworks has been steadily improving Matlab plotting functionality. Since Matlab R2020a, Matlab can directly create 3-D interactive plots of the Earth. Previously, it was necessary to export KML and use Google Earth for example to plot 3-D data with rendering of the Earth’s surface, so it’s nice to have the option to avoid those additional steps with external programs.

The plotting backend is WebGL, which has been used by Matlab since R2016a in uifigure. Matlab WebGL uses the GPU, including non-discrete GPUs built into the CPU as used in basic laptop PCs. The animation is generally fluid for smaller plots. With non-discrete GPU and maximized (full screen) globe plots, the GPU utilization may hit 100%, reducing the interactivity smoothness.

Caveat

At this time, it appears that exportgraphics is unable to capture 3D geoglobe plot to disk–legacy functions like print() and saveas() also don’t work. This is despite exportgraphics() working for general non-geoglobe uifigures. This is a fairly serious limitation for programatically saving animated 3D data. Reach out to the Mathworks to let them know if this is an issue for you as well.

Example

Experiment with geoplot3 and checkout the geoglobe gallery to generate a wide variety of intuitive, appealing and animated 3D plots.

fg = uifigure;
geoglobe(fg)

Notes

In Python, plotting data over a 3-D rendered Earth surface requires additional libraries like Mayavi or IPyParaView that serve as a frontend to VTK.

Git verbose connection diagnostics

Git can use multiple transport protocols. We focus here on diagnosing connection issues for two commonly used Git transfer protocols: SSH and HTTPS.

Regardless of protocol used, Git trace capability can give more details. For example, before the problem commands, type:

  • Windows: set GIT_TRACE=1
  • MacOS / Linux: export GIT_TRACE=1

Git over HTTPS

Git clone over HTTPS can hang for a minute or more. Git clone from Overleaf will hang while other collaborators are making edits via the Overleaf web browser editor. Overleaf uses Git over HTTPS, and when Overleaf is busy (say, near federal grant deadlines) Git operations can be significantly slower on Overleaf. The verbose options of Git don’t add much information for Git over HTTPS connections, just a single additional line is added by --verbose like:

POST git-upload-pack (150 bytes)

HTTPS timeout

Git over HTTPS can have a timeout set like:

git config --global http.lowSpeedLimit 1000
git config --global http.lowSpeedTime 10

This means that connections of less than 1 kB / sec for 10 seconds will be dropped. This avoids endless hangs, but could cause marginal connections to fail, so adjust parameters based on your connection and server if needed.

Git over SSH

Git over SSH verbosity can be added on a per-command basis starting with:

git -c core.sshCommand="ssh -v"

This is quite verbose, so we usually wouldn’t add this to the .git/config file.

SSH timeout

Git over SSH timeouts can be addressed via SSH settings.

references

GitLab Git troubleshooting connection tips

Print PDF from Matlab

Saving PDF of plots from Matlab is an effective way to use Matlab plots in LaTeX or for general sharing of high quality plots. Matlab ≥ R2020a has exportgraphics, which solved many of the previously long-standing issues with getting production-ready plots from Matlab. We also recommend tiledlayout for creating subplots. The Mathworks is continuing to upgrade Matlab’s plotting capabilities, so keep an eye on the release notes for new releases, it may finally fix issues you’ve had in the past.

Example

fg = figure;
ax = axes('parent', fg);
plot(ax, 1:10)

exportgraphics(fg, 'line.pdf', 'Resolution', 300)

Obsolete

The “print” and “saveas” functions are generally not recommended for saving plots from Matlab as they suffer from matching the saved file to what is seen on the screen. Use “exportgraphics” instead.

  • print() allows controlling image DPI via the -rDPI option, while saveas() is fixed at 150 DPI
  • saveas() is an “easier” version of print(), since saveas() is actually making calls to print() for both Matlab and Octave.

Print PDF from Matlabplot

Saving PDF of plots from Matplotlib is an effective way to use Matplotlib plots in LaTeX or for general sharing of high quality plots. This can be accomplished by either

  • save to PDF from the Matplotlib figure GUI
  • use figure.savefig() to save to PDF from Matplotlib–simply use a filename with “.pdf” suffix.

Headless (fast) PDF generation

Where local display of figures is not desired, that is, to save figures to disk without displaying them onscreen first, which is generally significantly faster, first do:

import matplotlib
matplotlib.use('Agg')

The Agg Matplotlib backend is commonly used in web / cloud services.

Example

from matplotlib.pyplot import figure

fg = figure()
ax = fg.gca()
ax.plot(range(10))

fg.savefig('line.pdf', bbox_inches='tight', dpi=300)

Eliminate Python 3.8+ __pycache__ directories

Python 3.8 brought the capability to put Python cache files under __pycache__/ anywhere. Control this behavior by setting the PYTHONPYCACHEPREFIX environment variable.

Example

  • Windows: set environment variable PYTHONPYCACHEPREFIX to %temp%
  • MacOS: export PYTHONPYCACHEPREFIX=/tmp
  • Linux: export PYTHONPYCACHEPREFIX=/tmp

Workaround

Python older than 3.8 can avoid writing .pyc files anywhere by setting environment variable:

PYTHONDONTWRITEBYTECODE=1

LaTeX wrapfigure debugging

LaTeX wrapfig.sty \wrapfigure is a great way to save space, but has several quirks to workaround. Overleaf also notes the touchiness of wrapfigure. The typical failure mode is that the wrapped figure will not appear, and the document text is wrapped for a few pages with blank column. The LaTeX log doesn’t give an error, just a warning about a missing figure reference.

Workarounds

  1. Don’t have wrapfigure next to a list (enumerated or itemized).
  2. put a blank line before and after wrapfigure stanza in the document body

Example

For #2, in the .tex file have like

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur ut faucibus ex.

\begin{wrapfigure}{R}{0.5\textwidth}
\includegraphics{...}
...
\end{wrapfigure}

Donec sed commodo eros. Sed vehicula elit ac tincidunt lacinia.

Numpy with GCC 10 fix

GCC 10 revealed a bug that I fixed for Numpy 1.19.0, where Numpy internally did not correctly compare two-digit version numbers. To use GCC 10 with Numpy, including for use with f2py, it’s best to upgrade Numpy to at least version 1.19.0

Configure LaTeX PDF version

The PDF version output by LaTeX is set by \pdfminorversion and this also sets the newest PDF version that LaTeX can use via \includegraphics. If \pdfminorversion default or setting is too low, by default LaTeX only issues a warning like:

PDF inclusion: found PDF version <1.7>, but at most version <1.5> allowed

and the PDF-input graphic will be missing. The too-old default PDF version of LaTeX is increasingly becoming an issue as PDF creation tools are rightfully increasing their default PDF output version from the ca. 2003 PDF v1.5 specification.

Fix

To mitigate these issues, we do two things near the top of each project’s main LaTeX file:

  1. create an error if the current PDF version is too old

    \pdfinclusionerrorlevel=1
    
  2. set the default PDF version to ca. 2008 PDF v1.7

    \pdfminorversion=7
    

It’s OK for these PDF configuration lines to come after LaTeX directives.