Meson file download

Unlike CMake, Meson does not have built-in the ability to download any file. While this could perhaps also be done via a custom_target(), we do it via run_command() in meson.build.

meson.build

run_command('python', 'meson_file_download.py', url, zipfn, md5hash)

run_command('python', 'meson_file_extract.py', zipfn, outpath)

meson_file_download.py

#!/usr/bin/env python
from pathlib import Path
import requests
import hashlib
import argparse


def url_retrieve(url: str, outfile: Path, md5sum: str = None, overwrite: bool = False):
    outfile = Path(outfile).expanduser().resolve()
    # need .resolve() in case intermediate relative dir doesn't exist
    if outfile.is_file() and not overwrite:
        return
    outfile.parent.mkdir(parents=True, exist_ok=True)

    R = requests.get(url, allow_redirects=True)
    if R.status_code != 200:
        raise ConnectionError('could not download {}\nerror code: {}'.format(url, R.status_code))

    outfile.write_bytes(R.content)

    if md5sum:
        if not file_checksum(outfile, md5sum, 'md5'):
            raise OSError('hash mismatch: Failed to download {}'.format(outfile))


def file_checksum(fn: Path, hash: str, mode: str) -> bool:
    h = hashlib.new(mode)
    h.update(fn.read_bytes())
    return h.hexdigest() == hash


if __name__ == '__main__':
    p = argparse.ArgumentParser()
    p.add_argument('url', help='URL to file download')
    p.add_argument('outfile', help='filename to download to')
    p.add_argument('md5sum', help='expected MD5 hash', nargs='?')
    P = p.parse_args()

    url_retrieve(P.url, P.outfile, P.md5sum)

meson_file_extract.py

#!/usr/bin/env python
from pathlib import Path
import argparse
import zipfile


def extract_files(zipfn: Path, outpath: Path, overwrite: bool = False):
    outpath = Path(outpath).expanduser().resolve()
    # need .resolve() in case intermediate relative dir doesn't exist
    if outpath.is_dir() and not overwrite:
        return
    zipfn = Path(zipfn).expanduser().resolve()
    with zipfile.ZipFile(zipfn) as z:
        z.extractall(str(outpath))


if __name__ == '__main__':
    p = argparse.ArgumentParser()
    p.add_argument('zipfile', help='.zip file to extract')
    p.add_argument('outpath', help='path to extract into')
    P = p.parse_args()

    extract_files(P.zipfile, P.outpath)

Switch from Python urllib.urlretrieve to requests for better features

Python’s old urllib.request.urlretrieve was deprecated several years ago. One of the biggest problems with urlretrieve is it doesn’t have a way to handle connection timeouts. This can lead to user complaints where they think your program is hanging, when really it’s a bad internet connection since urlretrieve will hang for many minutes.

Fix with requests

Python requests is strongly recommended for most Python internet use. Instead of bashing your head against solved problems with the vagaries of network and internet connections, use requests.

This is a recommended, robust way to download files in Python with timeout. I name it url_retrieve to remind myself not to use the old one.

from pathlib import Path
import requests

def url_retrieve(url: str, outfile: Path):
    R = requests.get(url, allow_redirects=True)
    if R.status_code != 200:
        raise ConnectionError('could not download {}\nerror code: {}'.format(url, R.status_code))

    outfile.write_bytes(R.content)

Why isn’t this in requests? Because the Requests BDFL doesn’t want it

Python asyncio.run boilerplate for Python >= 3.5

Concurrency is builtin to Python via asyncio. Python 3.6 added AsyncIO Generators, which greatly streamline asynchronous algorithms best expressed with generators. The async for introduced in Python 3.5 also simplifies expression of asynchronous for loops.

As in Julia, the expression of asynchronous structures in Python does not implement concurrent execution. Concurrent execution in Python is governed by collections of tasks or futures such as asyncio.gather and initiated by a runner such as asyncio.run

However, asyncio.run doesn’t currently handle all use cases across operating systems. AsyncIO subprocess in particular needs special options when using Python 3.6 or 3.7. The options needed are not the same for every project, depending on the asynchronous functions used. To support the use of asyncio.run() like behavior for Python ≥ 3.5 while also supporting current Python versions, consider the boilerplates below.

asyncio.subprocess

A complete single-file example that works for Python ≥ 3.5 is date_coro.py. This supports AsyncIO subprocess.

def runner(fun, *args):
    """
    Generic asyncio.run() equivalent for Python >= 3.5
    """
    if os.name == 'nt' and (3, 7) <= sys.version_info < (3, 8):
        asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())  # type: ignore

    if sys.version_info >= (3, 7):
        result = asyncio.run(fun(*args))
    else:
        if os.name == 'nt':
            loop = asyncio.ProactorEventLoop()
        else:
            loop = asyncio.new_event_loop()
            asyncio.get_child_watcher().attach_loop(loop)
        result = loop.run_until_complete(fun(*args))
        loop.close()

    return result

asyncio.open_connection

For asyncio.open_connection, the ProactorEventLoop makes a lot of warnings with trapped RuntimeError. The workaround for this is to not use the ProactorEventLoop. This runner.py works for Python 3.5..3.8 at least.

def runner(fun, *args):
    """
    Generic asyncio.run() equivalent for Python >= 3.5 that
      does not use ProactorEventLoop on Windows
    """
    use_run = ((os.name == 'nt' and (3, 8) > sys.version_info >= (3, 7)) or
               (os.name != 'nt' and sys.version_info >= (3, 7)))

    if use_run:
        result = asyncio.run(fun(*args))
    else:  # 3.8, 3.6, 3.5
        loop = asyncio.SelectorEventLoop()
        result = loop.run_until_complete(fun(*args))
        loop.close()
    return result

Running Matlab from Pytest

A software package may have Matlab and Python functions. Whether or not the Matlab code calls Python functions, Matlab code can be included via Pytest. To test the Matlab functions from Pytest, create a test_matlab.py file like:

from pathlib import Path
import subprocess
import pytest
import shutil

R = Path(__file__).parent

OCTAVE = shutil.which('octave-cli')
MATLAB = shutil.which('matlab')


@pytest.mark.skipif(not MATLAB, reason="Matlab not available")
def test_matlab_api():
    if subprocess.run([matlab, '-batch', 'exit(isempty(pyversion))'], timeout=60).returncode:
        pytest.skip('Python not setup in Matlab')

    subprocess.check_call([MATLAB, '-batch', 'test_myscript'],
                          cwd=R, timeout=60)


@pytest.mark.skipif(not OCTAVE, reason='octave not found')
def test_octave_api():
    subprocess.check_call([OCTAVE, 'test_myscript.m'],
                          cwd=R, timeout=60)


if __name__ == '__main__':
    pytest.main(['-xrsv', __file__])

This assumes the Matlab test script resides in the same directory as the test_matlab.py file, and is named like test_api.m.

Git global recursive directory ignore

Certain IDEs like PyCharm create per-project cache directories with metadata relevant to the IDE configuration for that project. For example, PyCharm creates .idea/ directory for each project. You could edit the .gitignore file for each repository. However, I prefer to ignore files in Git system-wide for all repos as follows:

Tell Git where the global ~/.gitignore file is:

git config --global core.excludesfile ~/.gitignore

Create / edit the ~/.gitignore file to include things like:

.idea/
.DS_Store
pip-wheel-metadata/

Access Windows Subsystem for Linux files from Windows

Related: Mount external drives in WSL


Since Windows 10 version 1903, it is possible to safely access WSL filesystem from Windows via a 9P server. This currently means the WSL distro of interest must be running for the Linux files to be accessible from Windows.

The WSL files are available from Windows on path like:

\\wsl$\Ubuntu\home\username

for an Ubuntu home directory, for example.

To keep things simpler and more consistent to use, we still recommend keeping files that need to be accessed from WSL and Windows under the usual Windows file system, making softlinks in WSL as useful.

For example, if I keep code in Windows under c:/users/username/code then in WSL I do one-time:

ln -s /mnt/c/users/username/code ~

Raw WSL files

Windows Subsystem for Linux places files for each WSL image uniquely named like:

WSLWindows
~%LOCALAPPDATA%\Packages\CanonicalGroupLimited.UbuntuonWindows*\LocalState\rootfs\home
/ %LOCALAPPDATA%\Packages\CanonicalGroupLimited.UbuntuonWindows*\LocalState\rootfs

Do not create/edit files from Windows in the “APPDATA” directories as it will corrupt the WSL filesystem.

Notes

Microsoft says don’t edit/write WSL files from Windows.

Ticwatch Pro review

A key feature of the TicWatch Pro is the dual-display monochrome LCD over the traditional OLED color display. The LCD contrast is less than traditional LCDs, but is visible in most lighting conditions. The LCD does not compromise the OLED brightness or clarity.

LTE

It’s important to distinguish between the slower 2018 TicWatch Pro (without LTE) with 512MB of RAM, and the significantly faster TicWatch Pro LTE with 1GB RAM. It’s not yet known if the TicWatch Pro will be carrier-agnostic as the eSIM should allow. The LTE bands supported by the TicWatch Pro LTE are 2, 4, 5, 12, 13, 17 that include TMobile, AT&T, Verizon and Sprint among others.

  • TicWatch Pro LTE FCCID: 2AP42-WF11026

Battery Bar levels

The five-segment LCD battery level indicator is currently programmed to these percentage ranges:

  • 90-100%: 5 bars
  • 70-89%: 4 bars
  • 50-69%: 3 bars
  • 30-49%: 2 bars
  • ≤ 29%: 1 bar

saving battery

The 2-3 day battery life is the “killer” feature of the TicWatch Pro. An important setting for battery life on any Wear OS watch is under Settings → Gestures → Touch-to-Wake Turning Touch-to-wake off gives 10+% battery life extension by avoiding the OLED watch display turning on inadvertently or on messages–except if you tip/touch the watch within a couple seconds of the vibrate notification.

Heart rate monitor background readings

The TicWatch Pro background heartrate monitor doesn’t work with Google Fit directly. Workarounds indicated on the forums included using the Mobvoi fitness app to sync with Google Fit. Despite indications from Mobvoi that this would be fixed by April 2019, Google Fit still doesn’t automatically monitor heartrate on the TicWatch Pro.

two buttons vs. three

Previously I used the LG Watch Sport W280 with three buttons. I was concerned about “downgrading” to two buttons and losing a rotatable crown. However, I soon enjoyed not having accidental screen turnons from the center button, and the double-tap and long-press button gestures give back most of the functionality of the missing third button.

Wear OS LTE smartwatches

An important smartwatch usability factor is whether the smartwatch has LTE. In 2019, the leading Wear OS watch with LTE in North America is the Ticwatch Pro LTE. Watches that are locked to Verizon Wireless LTE bands are less interesting to many users because they may want to use a different prepaid LTE provider or use the watch internationally.

Google Fi and Ting work with Wear OS smartwatches having LTE via the T-Mobile 2G / 3G / 4G cellular network. AT&T also works with Wear OS LTE smartwatches. The overlapping T-Mobile / AT&T LTE bands with Wear OS include bands 2, 4, 5, 12 and 17.

Wear OS LTE smartwatch comparison

ModelCPULTE BandsGoogle Fi/Ting
Ticwatch Pro LTESD21002,4,5,12,13,17eSIM (TBD)
LG Watch Sport LG-W280ASD2100 MSM8909W2,4,5,13Yes
LG Urbane 2nd Ed. LTE LG-W200ASD 4002, 5Yes
LG Urbane 2nd Ed. LTE LG-W200VSD40013No (wrong bands)
Verizon Wear24SD210013No (wrong bands)
Huawei Watch 2SD21001,3,7,8,39,41No (wrong bands)

Why are LTE bands important for smartwatches?

The LTE bands are important for knowing if the watch would work on a particular carrier. Sometimes a watch has LTE bands the carrier doesn’t and vice versa. As long as one or more LTE bands overlap, your watch might work–always double check before making a purchase! Carriers don’t use all their authorized LTE bands throughout their entire network. Some LTE bands are used only in urban areas.

Fix Git line endings on Windows + Cygwin or WSL

When using Git on Windows with Cygwin or Windows Subsystem for Linux, CRLF conflicts can falsely make your Git repo “dirty”. That is, git diff from Cygwin or WSL will show ^M at the end of each line and not let you merge code on git pull. This wastes developer time and possibly causes missed code changes or needless commits. We suggest to force LF line endings no matter what environment the user is in. Even Windows Notepad supports LF line endings.

Fix

Create in each Git repo a file .gitattributes including:

DO NOT just use * as that can goof up binary files in your repo.

git config --global core.autocrlf input

git config --global core.eol lf

This tells Git to force line endings \n on committed files.

Raspberry Pi power stability notes

Related: Measure Raspberry Pi CPU temperature


The Raspberry Pi 3+ and 4 DC power input is handled by the MaxLinear MXL7704 power management IC (PMIC). A yellow lightning bolt is GPU-superimposed on the Raspberry Pi display output for low voltage. In general computing platform operation is not guaranteed with voltages out of tolerance. The SD card can become unreadable, the Pi may have random malfunctions, and corrupt data (bad writes) on the SD card. 2.5 Amps is the minimum recommended Raspberry Pi power supply current rating, but 3.0 Amps is better if using peripherals (hats). The cable between the power supply and Raspberry Pi must be of good quality to minimize voltage drop and unstable operation.

The Raspberry Pi 4 power connector is USB-C, which is markedly more robust than the micro-USB power connector of legacy Raspberry Pi models. However, it has been acknowledged that the first revision of the Raspberry Pi 4 has a defective USB-C implementation that won’t work with some standard USB-C cables.

Power Management IC

The APX803 is a simple DC input voltage monitoring IC. Raspberry Pi model 1, 2 and 3 used custom circuitry instead of a COTS PMIC to handle sequencing of discrete DC power input to the Pi subsystems. The significant current demands of the Raspberry Pi 3+ and 4 helped drive the choice of the MaxLinear MXL7704.

It has been noted that damaging the PMIC can make the Raspberry Pi too difficult to repair.

low voltage alarm

DC Voltageicon
> 4.65none
< 4.65yellow lightning bolt

Measure DC input

There is no built-in capability to measure the Raspberry Pi DC input voltage without adding an external ADC. It is possible to read the input voltage state (OK/low) from the Terminal:

Notes

When compiling programs with concerns about excessive power consumption, consider not compiling in parallel. For make, make -j2 uses 2 CPU cores for example–omit the -j option to compiler with 1 CPU core.