Put Git revision in executable or library

Traceability of a binary artifact such as an executable or library can be improved by writing information about the Git repository status into the artifact itself. This is a finer-grained implementation of the version number we are accustomed to seeing in the command line interface of executables. This example doesn’t cover every possible thing to be traced, for example non-version controlled artifacts that are linked in. This example just covers the Git repo of the current CMake project. Nonetheless, those needing more advanced traceability can build upon this example.

Usually for easier reuse across projects, we put this in a separate CMake script file like gitrev.cmake and include it from the main CMake project.

See the example in Fortran2018-examples repo.

Get Matlab HDF5 version

Matlab upgraded to HDF5 1.8.12 in R2015a. Matlab R2020b uses HDF5 1.8.12. HDF5 1.8.12 was released in November 2013. HDF Group official support for HDF5 1.8 ends in 2021.

Check Matlab HDF5 library version by:

[major,minor,rel] = H5.get_libversion()

Compatible HDF5 versions

Newer versions of the HDF5 library can write HDF5 1.8 files when specific options are used. This may be necessary to allow Matlab to read HDF5 files written by other applications.

Append PATH in GitHub Actions

One can globally set environment variables in GitHub Actions by using env: at the top level of a “.github/workflows/ci.yml” file, for example:

name: ci

env:
  CMAKE_GENERATOR: Ninja
  CC: gcc

In other cases like appending to PATH, this must be done dynamically but can still have global scope. In GitHub Actions this is done by writing to environment files.

Prepend PATH

To add “~/.local/bin” to PATH, under a run: stanza per operating system:

Linux / MacOS

echo "${HOME}/.local/bin" >> $GITHUB_PATH

Windows

Windows defaults to PowerShell, so the syntax is distinct from Unix shells:

echo "${HOME}/.local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Matlab websave SSL certificates

Matlab websave allows specifying details options to control HTTP behavior via weboptions. Typical options that are modified include Timeout and SSL Certificate checking bypass. While SSL certificate checking adds security to web operations, some HPC systems have old or broken certificates. Other systems may simply need environment variable SSL_CERT_FILE set to tell Matlab’s vendored cURL where the cert file is.

As a last resort, certificate checking can be turned off, but this opens up code / file integrity and concomitant security issues.

Configuration

A generally better solution than disabling certificate checking is to configuration your user profile to tell cURL and Git the location of the system certificates. For this example we assume the certificate file is at “/etc/ssl/certs/ca-bundle.crt”.

cURL SSL config

set environment variable by editing ~/.bashrc

export SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt

This for example can fix issues with Matlab websave() that uses Matlab’s vendored cURL.

Git SSL config

Tell Git where the cert file is by:

git config --global http.sslCAInfo /etc/ssl/certs/ca-bundle.crt

Example

This example sets timeout to 15 seconds and specifies custom SSL cert location when environment variable SSL_CERT_FILE is set.

if isfile(getenv("SSL_CERT_FILE"))
  web_opts = weboptions('CertificateFilename', getenv("SSL_CERT_FILE"), 'Timeout', 15);
else
  web_opts = weboptions('Timeout', 15);
end

websave(saved_file, url, opts);

Matlab on Travis-CI

Matlab on Travis-CI is useful to automatically unit test Matlab code on each git push as with other coding languages. As typical for CI, we use a “.travis.yml” file in the Git repo to control Travis-CI behavior.

language: matlab

matlab:
- latest

script:
- matlab -batch "r = runtests('IncludeSubfolders',true); assert(~isempty(r)); assertSuccess(r)"

Matlab CI with compiled program

Using Matlab with code from compiled languages takes a few extra parameters. Use a modern Ubuntu version compatible with the Matlab version to get more modern library versions when we use compiled code with the Matlab tests.

dist: bionic
# Matlab R2020a libstdc++ needs Ubuntu 18.04

language: matlab

matlab:
- latest

git:
- depth: 3
- quiet: true

addons:
  apt:
    packages:
    - gfortran
    - libhdf5-dev
  snaps:
  - name: cmake
    confinement: classic

before_script:
- export PATH=/snap/bin:$PATH

script:
- matlab -batch "r = runtests('IncludeSubfolders',true); assert(~isempty(r)); assertSuccess(r)"

Here we assumed the compiled project code for Matlab uses CMake, Fortran and HDF5.

Notes

For private projects that require offline CI, solutions include GitHub Actions self-hosted runner or self-hosted AppVeyor

Python asyncio.run boilerplate

Concurrency is built into Python via asyncio. AsyncIO generators are implemented with yield much like synchronous generators. async for also simplifies expression of asynchronous for loops.

As in Julia, the expression of asynchronous structures in Python does not implement concurrent execution. Concurrent execution in Python is governed by collections of tasks or futures such as asyncio.gather and initiated by a runner such as asyncio.run

asyncio.run() doesn’t handle all use cases. AsyncIO subprocess may need specific asyncio loop configuration. The options needed are not the same for every project, depending on the asynchronous functions used.

asyncio.subprocess

The example date_coro.py uses AsyncIO subprocess, which needs ProactorEventLoop on Windows in project __init__.py:

import os
import asyncio

if os.name == "nt":
    asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

asyncio.open_connection

For networking apps using asyncio.open_connection the ProactorEventLoop emits numerous warnings with trapped RuntimeError. A workaround is to not use the ProactorEventLoop. Do this by adding to project __init__.py:

import os
import asyncio

if os.name == "nt":
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Matlab package import like Python

Matlab users can share code projects as toolboxes and/or packages. Matlab packages work for Matlab ≥ R2008a as well as GNU Octave. Matlab toolboxes work for Matlab ≥ R2016a and not GNU Octave. The packages format brings benefits to toolboxes as well.

Matlab namespaces

A key issue with Matlab vs. Python arise from that Matlab users often add many paths for their project. If any function names clash, there can be unexpected behavior as it’s not immediately clear which function is being used without further investigation of path ordering. As in Python and other languages, there is considerable benefit for using a package format where the function names are specified in their namespace.

addpath example

To enable Matlab package format, we explain by example. Suppose a project directory structure is like:

myproj
  utils
    mem1.m
  conversion
    deg1.m
  sys
    disk1.m

To use these functions, the end users do:

addpath(genpath('myproj'))

This is where the namespace can have clashes, and with large projects it’s not clear where a function is without further introspection.

package example

To make this project a Matlab / Octave package, change the subdirectories containing .m files to start with a “+” plus symbol:

myproj
  +utils
    mem1.m
  +conversion
    deg1.m
  +sys
    disk1.m

Now the end users will simply:

addpath('myproj')

and then access specific functions like:

myproj.utils.mem1(arg1)

Then multiple subdirectories can have the same function name without clashing in the Matlab namespace. Suppose the function “mem1” is used frequently in another function. To avoid typing the fully resolved function name each time, use the import statement:

function myfunc()

import myproj.utils.mem1

mem1(arg1)

mem1(arg2)

Private functions

Matlab packages can have private functions that are only accessible from functions in that level of the namespace. Continuing the example from above, if we added function:

myproj
  +utils
    private
      mysecret.m

then only functions under +utils/ can see and use mysecret.m function. mysecret() is used directly, without import since it’s only visible to functions at that directory level.

Matlab toolbox .mltbx

Matlab .mltbx toolboxes became available in R2016a. The Matlab-proprietary toolbox format also allows end users to create their own packages containing code, examples and even graphical Apps. In effect .mltbx provides metadata and adds the package to the bottom of Matlab path upon installation. The installation directory is under (system specific)/MathWorks/MATLAB Add-Ons/Toolboxes/packageName. Whether or not the project uses .mltbx, the namespace of the project is kept cleaner by using a Matlab package layout.

Running Matlab and GNU Octave via Pytest

A software package may have Matlab and Python functions. Plain Matlab code can be included via Pytest for Matlab and GNU Octave. To test the Matlab functions from Pytest, create a test_matlab.py file like:

from pathlib import Path
import subprocess
import pytest
import shutil

R = Path(__file__).parent

OCTAVE = shutil.which('octave-cli')
MATLAB = shutil.which('matlab')


@pytest.mark.skipif(not MATLAB, reason="Matlab not available")
def test_matlab_api():

    subprocess.check_call([MATLAB, '-batch', 'assertSuccess(runtests)'],
                          cwd=R, timeout=60)


@pytest.mark.skipif(not OCTAVE, reason='octave not found')
def test_octave_api():

    subprocess.check_call([OCTAVE, 'test_api.m'],
                          cwd=R, timeout=60)

This assumes the Matlab test script resides in the same directory as the test_matlab.py file, and is named like test_api.m.

NOTE: Be sure that Matlab “runtests()” actually picks up the desired tests. We do not want the Matlab output to say

Totals:
   0 Passed, 0 Failed, 0 Incomplete.
   0 seconds testing time.

Notes

Generally using __file__ is not robust for packages but we might relax that restriction for test files where a clear error will result.

Fix Fitbit not syncing

The Fitbit app on certain phones with certain devices can sometimes fail to sync. This might happen multiple times a month. To workaround this issue, try force stopping the Fitbit app. This normally shouldn’t lose any data, and reopening the Fitbit app and dragging down should re-sync in a few seconds.

Also, check that Bluetooth is enabled on the Android device.

Finally, try rebooting the phone and watch if the previous steps don’t help.

Force stop Android app

Force stop an Android app by:

Settings → Apps and Notification → Sell all apps → scroll to app name and click → click Force Stop