Intro-ML-Arts-IMA-F24

Transformers.js with p5.js

• SLIDES - Thank you @xenova!

Objectives

• 🎯 Understand what Transformers.js is and its applications for creative projects.
• 🔌 Learn how to integrate an ES6 module like Transformers.js into p5.js using dynamic imports.
• ⏳ Gain experience with asynchronous JavaScript (async/await) for handling machine learning models.
• 🤖 Explore a selection of pre-trained models available in Transformers.js (e.g., sentiment analysis, object detection, Whisper, LLMs).
• 🚀 Learn about the new WebGPU support and how to use it for accelerated machine learning.

Transformers.js Guides and Documentation

• 📚 Transformers.js Documentation
• 💻 Transformers.js GitHub Repository
• 📰 Transformers.js v3: WebGPU Support, New Models & Tasks, and More…
• 🔍 Transformers.js Examples and Demos
• 📦 Official npm package

JavaScript for Transformers.js

• 🔄 JavaScript Asynchronous Programming: Async/Await
• 📦 JavaScript Dynamic Imports
• ⚙️ WebGPU Overview
• 🚂 Videos on promises, async/await, and try/catch
• 🚂 Video on arrow functions
• 📚 toDataURL()
• 📚 Object and Array Destructuring
• 🚂 let vs. const

p5.js DOM Basics and Resources

• 🌐 Beyond the Canvas, p5.js tutorial
• 🚂 What is HTML? (video)
• 🚂 What is CSS? (video)
• 🚂 Creating DOM Elements with p5.js (video)
• 🎨 CSS Property Reference

Example Projects and Demos

• 🎮 Doodle Dash with Transformers.js
• 🖼️ NYU Shanghai DURF research project on Image Segmentation with transformers.js

Code Examples

• Object Detection - Image
• Object Detection - Video
• Sentiment Analysis
• Depth Estimation - Image
• Depth Estimation - Video
• Text Generator

Assignment

For this week’s assignment:

1. Choose any model available in Transformers.js and create a p5.js sketch using it. Try any transformer.sj compatible model! Some things you might look for are:

   • Object Detection: 🕵️ Detect objects in an image.
   • Image Segmentation: ✂️ Segment an image into parts.
   • Text Generation: ✍️ Generate text based on a user prompt.
   • Whisper (Speech-to-Text): 🎙️ Transcribe audio from a file or real-time microphone.
   • Depth Estimation: 📏 Create a depth map from an image or video feed.

2. Compare and Contrast:

   • Compare your experience of using Transformers.js with ml5.js. Reflect on differences in ease of use, performance, model variety, and integration with p5.js.

3. Document your sketch in a blog post and add a link to the Assignment Wiki page. Remember to include visual documentation (screenshots, GIFs, etc) Also, please document what did and did not work! What errors did you encounter with Transformers.js?