Python-based Computer Vision in the Browser

Darren Wiens July 26, 2022

Prototyping and Python are big parts of what I do at Sparkgeo, so when PyScript was announced at PyCon US 2022, I was all ears. PyScript, along with WebAssembly and Pyodide, enables us to run Python directly in the browser, without the explicit need for our own APIs or servers. JavaScript (JS) will surely remain the dominant language in browsers for the foreseeable future. However, enabling Python developers to quickly create frontend applications with minimal friction is a powerful concept.

In this post, I will walk us through setting up a PyScript application, and adding Python-based computer vision (OpenCV) functionality to a web map, all running within the browser.

Setting up a PyScript app is as simple as creating an HTML file, importing JS and CSS files, and including Python code within a <py-script> HTML tag. To include the PyScript code and styles, add the following to the <head> section of your HTML file:

<link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
<script defer src="https://pyscript.net/alpha/pyscript.js"></script>

We can also configure PyScript setup options within a new <py-config> tag. As of the time of writing, we must use the development branch of Pyodide to successfully import OpenCV into our application. We can do so by including the following configuration in the <head> section of our HTML:

<py-config>
  - autoclose_loader: true
  - runtimes:
    - src: "https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js"
      name: pyodide-dev
      lang: python
</py-config>

There are several packages available for out-of-the-box import through Pyodide. You can view the list for the current release, v0.20.0, under the version tag on GitHub. Note that opencv-python, the package we need for OpenCV functionality, is not in the list. However, by switching to the dev version of Pyodide, we can access packages that will be available in upcoming releases, found in the main branch.

Third party packages are imported through the new <py-env> HTML tag. We can also add paths to external Python files – I like to use external Python files to take advantage of code formatting in my code editor.

<py-env>
  - opencv-python
  - paths:
    - ./main.py
</py-env>

Now, we can start writing Python within the new <py-script> HTML tag, alongside JS within the familiar <script> tag:

<body>
  <py-script>
    print(“This is Python!”)
  </py-script>
  <script>
    console.log(“This is JavaScript!”)
  </script>
</body>

Finally, we can import Python objects from external files:

<py-script>
  from main import click_py
  click_py()
</py-script>

Great, you’re ready to start making your own PyScript app!

I’ve posted an example PyScript application on GitHub. The application uses the Mapbox GL JS library to draw and interact with a map, and uses Python and OpenCV to draw computer vision features derived within the visible extent of the map.

Example application, showing image segmentation functionality through the Detect Visual Contours button.

A few notes on the sample application:

While PyScript provides a selection of pre-configured visual component tags, the example uses plain HTML buttons, implementing a sparsely documented PyScript feature that allows for additional HTML attributes that route events to Python. In the example, I use the pys-onClick event to send click events directly to Python functions.
The JS code includes map creation, map interaction, and two functions called from Python for their returned values (i.e. the map and the access token). There is likely a way to port the entirety of the JS code to Python, but that is an exercise for the future.
The Python code includes the click event handlers, a function to retrieve an image from the Mapbox Static Images API, and code to apply a few basic OpenCV functions to the static map image.
It is important to note how values are passed between Python and JS (excellent resource: John Hanley’s PyScript: JavaScript and Python Interoperability). The basic premise is that JS functions can be called from PyScript through the js object. If we want to retrieve a value from JS, we can create a JS function that returns the value (e.g. getToken()). If we want to execute a JS function passing values from PyScript, we can create a JS function that accepts the parameter (e.g. updateCorners(jsonData)).

If you deploy the application (after configuring your own Mapbox Access Token), you should see a map, and clicking on the buttons will draw new, computer-vision derived features.

That’s it! I’d be interested to know if you extend the sample application, and otherwise how you use PyScript in a geospatial context. Please get in touch!