Web GL + p5.js for Google Summer of Code 2017

I spent the summer improving the p5.js WebGL mode with Stalgia Grigg, as part of Google Summer of Code. With the support of our mentors, Lauren McCarthy, David Wicks, and Dan Shiffman, we were able to implement some new features and make some performance + architecture improvements to the library.

We kept a shared branch on github, WebGL_GSOC17, to compile changes that Stalgia and I were both making to the project (if you want to see my changes in isolation, you can do so here). We were given a lot of agency to shape the library and to choose what features we worked—part of the process was figuring out what was needed and most urgent. (A couple last-minute pull requests are here and here, but will be merged to our main working branch soon.)

Planning

Stalgia initiated a planning document for us to track open issues and feature requests alongside our own observations about what was needed in the p5.js WebGL feature set. As we were learning the webgl part of the codebase, we kept notes here, tracking things that were hard for us to follow as new contributors.

Looking at our todolist, it became clear that we couldn’t tackle all of the feature requests/enhancements and architecture improvements in just the summer. We made another document detailing goals and priorities for the WebGL module to outline some higher level goals for the project. We decided to focus on architecture, performance, and feature improvements that would be difficult for a contributor to take on without an extended period of time to focus on the project.

Writing Code

I developed a few new abstractions to add to the WebGL library. Code having to do with shaders and textures was modularized moved into its own files, making it easier to understand the WebGL workflow and to optimize performance. Organizing the code made it easier to understand when the code was doing extra work to set up the GL state and made it possible to do some optimization of rendering 3D shapes. A short description of each new object is below—view the developer documentation on the p5.js wiki for more information.

p5.Shader

The p5.Shader object provides access to the uniforms and attributes of a GL program, compiles and links the program, and provides an API to set up the shader program for rendering.

To give the new shader objects a good test, I added some minimal support for custom shaders via the loadShader() method, as a Processing user might expect. It’s not quite as sophisticated or fault-tolerant as its Processing counterpart yet, but the architecture withstood the test!

Screenshot of sketch running in the browser with a grid of cubes. The surface of each cube is rendered with a custom shader that blends an image (depicting the u,v texture grid) with a playing video (depicting hand gestures), and tinted blue/pink according the user's mouseX and mouseY position.

The screenshot above shows a custom shader rendering the surfaces of a grid of cubes. The shader blends an image and video using two different texture samplers in GLSL and tints the color according to mouseX and mouseY via custom uniforms.

p5.Texture

The p5.Texture object handles converting various format (images and videos) to webgl textures.

Performance

With new abstractions in place for shaders and textures, it was easier to track down performance bottlenecks in the WebGL code. Previous slowdowns had been made because it was difficult to deduce context, so instead, the code often did more work than it needed to (for example, the first bug I fixed this summer was speeding up calls to begin/endShape, which compiled and linked a new shader every single frame).

Screenshot of sketch running at 50 frames per second while drawing 1000 randomly placed and filled triangles.

At the start of the summer, the sketch depicted above—with 1000 randomly placed and filled triangles—would have been impossible to run because the performance hit was so severe! Mid-way through the summer, it ran at ~45 frames/second, and is now up to ~50 frames/second.

Camera and Coordinate System

Taking the lead from Processing, I adjusted the coordinate system to transform the GL projections to use a left-handed coordinate system for consistency with the library’s 2D mode (the idea is that one can add the WEBGL parameter to any sketch and expect things to look almost the same). The existing code had a simplified camera model that needed to be expanded to add new features and fix the coordinate system.

Documentation

See the WebGL Module Architecture page on the p5.js wiki for developer documentation of new features added to the library!

Future Work

See Open issues for p5 + WebGL.

Text Rendering

The new p5.Texture API was designed with a future implementation of the text() feature in mind. Using this article as a guide, I’d like to implement texture-based rendering of text in webgl mode.