Overloading functions in Matlab and GNU Octave

Matlab and GNU Octave are constantly adding new functionality. However, legacy versions remain in use for years. One in general may not be able to overload a Matlab function in Octave because of how Octave provides many functions as files, and cannot distinguish them from user files.

The “overloading” we do here is to provide a similarly but distinctively named function that seamlessly uses the new syntax when available, with a simple fallback. In this isfile example that was added in Matlab R2017b, the fallback only works for single files.

isfile fallback

Create a file “is_file.m” containing:

function ret = is_file(path)

if exist('isfile', 'builtin') == 5 || exist('isfile', 'file') == 2
  ret = isfile(path);
else
  ret = exist(path, 'file') == 2;
end

end % function

Using Ruby Gem install with GitHub Actions

To use Ruby quickly and easily in GitHub Actions, add this YaML snippet in your Job:

    - uses: actions/setup-ruby@v1
      with:
        ruby-version: '2.x'

Example

A complete job (named integration) example where Ruby packages are called from Python is below. Example from Python Linguist.

  integration:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-python@v1
      with:
        python-version: '3.x'
    - uses: actions/setup-ruby@v1
      with:
        ruby-version: '2.x'
    - run: gem install github-linguist --no-document
    - run: pip install -e .[tests]
    - run: pytest -r a -v

Pytest ignoring Meson subprojects

Meson projects may contain Python code, including Meson subprojects. However, the Meson subproject code may not be relevant to the top-level Meson project Python code. Then, Pytest Python test suites may fail when the subprojects/ directory tree is searched and unwanted tests are run.

Ignore directories with Pytest

While this example is for Meson subprojects, it is obviously applicable to many other Python projects.

Create pytest.ini file at the top-level project directory, containing:

[pytest]
norecursedirs = subprojects .* build dist CVS _darcs {arch} *.egg venv

Install Matlab Engine API in Python


Matlab Engine API allows calling Matlab functions from Python code.

Install

Find the directory Matlab is installed in by from Terminal / Command Prompt:

matlab -batch matlabroot

Assuming Matlab root is like MATLAB/R2019b then do from Terminal / Command Prompt:

cd MATLAB/R2019b/extern/engines/python
  • Mac or Linux:

    python setup.py build --build-base=$(mktemp -d) install
  • Windows Powershell:

    python setup.py build --build-base=$env:temp install

Permissions Error

If error about needing root permissions, you’re probably installing in system Python. We normally recommend using Anaconda Python instead. If you must use system Python, add the --user flag to the command above.

Examples

Python → Matlab

import matlab.engine
eng = matlab.engine.start_matlab('-nojvm')
y=eng.asin(1.)

eng.quit()

The Matlab Engine should take less than 1 second for Matlab Engine to start when called from Python.

matlab.double → Numpy

enclose the matlab.double variable in numpy.asarray()

Python float → Matlab Engine

Python floats pass into Matlab Engine by including a period . after the number.

  • asin(1) fails
  • asin(1.) works

numpy.ndarray → Matlab Engine

Matlab Engine from Python can pass N-dimensional arrays.

Asynchronous Matlab Engine

Matlab Engine provides asynchronous call with background=True option  

Notes

Running Matlab and GNU Octave via Pytest

A software package may have Matlab and Python functions. Plain Matlab code can be included via Pytest for Matlab and GNU Octave. To test the Matlab functions from Pytest, create a test_matlab.py file like:

from pathlib import Path
import subprocess
import pytest
import shutil

R = Path(__file__).parent

OCTAVE = shutil.which('octave-cli')
MATLAB = shutil.which('matlab')


@pytest.mark.skipif(not MATLAB, reason="Matlab not available")
def test_matlab_api():
    if subprocess.run([matlab, '-batch', 'exit(isempty(pyversion))'], timeout=60).returncode:
        pytest.skip('Python not setup in Matlab')

    subprocess.check_call([MATLAB, '-batch', 'test_api'],
                          cwd=R, timeout=60)


@pytest.mark.skipif(not OCTAVE, reason='octave not found')
def test_octave_api():
    subprocess.check_call([OCTAVE, 'test_api.m'],
                          cwd=R, timeout=60)


if __name__ == '__main__':
    pytest.main(['-r', 'a', '-v', __file__])

This assumes the Matlab test script resides in the same directory as the test_matlab.py file, and is named like test_api.m.

Import Python user modules in Matlab

This page shows how to setup Matlab with user Python install (e.g. Anaconda Python) on Linux, Mac or Windows. It goes without saying, the Python program must work first with Python alone for it to work in Matlab. For concurrent Python modules using asyncio, you may need to create a shim function to allow Matlab to call the Python module.

Install Python for Matlab

Matlab is designed to work with specific Python versions for each Matlab version. Matlab will not specifically tell you when you’re using an incompatible Python version, but you may get unstable operation or errors. Matlab R2019b introduced the pyenv Python environment manager.

The Python executable is found from Terminal / Command Prompt:

python -c "import sys; print(sys.executable)"

For Windows, it may be like C:/Miniconda3/python.exe and for MacOS / Linux it may be like ~/Miniconda3/python.

This Matlab command is persistent–Matlab remembers this Python choice even after restarting Matlab.

pyenv('Version', 'C:/Miniconda3/python.exe')

Verify Matlab → Python config by typing pyenv from within Matlab.

Notes

  • Python module import is implicit in the Matlab → Python module function call. There is no need to import numpy etc. from Matlab
  • Python executable choice persists across Matlab sessions–Matlab “remembers” even after you restart Matlab or reboot the computer.
  • editing imported Python module code requires restarting Matlab to take effect.

Older versions of Matlab back to R2014b can also call Python from Matlab, but is more difficult to use, particularly for arrays / matrices. Matlab ≥ R2018b is strongly recommended for use with Python.

Force apt to IPv4

Sometimes a remote host or your ISP is temporarily misconfigured with regard to IPv6. In this case, apt update can simply hang, for example:

Connecting to mirrordirector.raspbian.org (2a00:1098:0:80:1000:75:0:3) Connecting to archive.raspberrypi.org (2a00:1098:0:80:1000:13:0:5)

You can temporarily force apt to use IPv4 with apt option -o Acquire::ForceIPv4=true.

example

apt -o Acquire::ForceIPv4=true update

apt -o Acquire::ForceIPv4=true upgrade

Numpy can't read .zip files

ZIP files or GZ files and the like can be quick-and-dirty ways to compress individual data files for retrieval from remote sensors. In particular, the GeoRinex program has extensive capabilities for transparently (without extracting to uncompressed file) reading .zip, .z, .gz, etc. compressed text files, which benefit greatly from storage space savings. It was surprising to find that transparently processing similarly compressed binary data is not trivial, particularly with numpy.fromfile. Numpy has unresolved bugs with numpy.fromfile that preclude easy use with inline reading via zipfile.ZipFile or tarfile. Specifically, the .fileno attribute is not available from zipfile or tarfile, and numpy.fromfile() relies on .fileno among other attributes.

numpy.frombuffer is not generally suitable for this application either, because it does not advance the buffer position. We are not saying there’s no way around this situation, but we chose a more generally beneficial path.

Use HDF5

When raw data files need to be compressed and then later analyzed, we use HDF5. Even when the original program writing the raw binary data cannot be modified, a simple post-processing Python script with h5py reads the raw data and converts to lossless compressed HDF5 on the sensor. Then, when the data is analyzed out-of-core processing can be used, or at least the whole file doesn’t have to be read to retrieve data from an arbitrary location in the HDF5 file. This allows getting nearly all of the size and speed advantages of HDF5 without modifying the original program.

Google Shared Drives benefits and caveats

Google Shared Drives ( formerly Google Team Drives) allows creating arbitrary numbers of shared drives consisting of people with Gmail addresses (corporate or personal). However, public folder Shared Drives are not possible as it is with personal Google Drive. Public individual file sharing is possible with Google Shared Drives, but it would be tedious to share very many links.

The alternative workaround is to add people you want to share files with to a big Shared Drive with appropriate permissions.

No granular folder permissions

Permissions are uniform across all folders & files in each Google Shared Drive. To workaround this limitation, create multiple team drives–even if it’s for the same physical team.

Notes