Handsfree.js

Build handsfree User Experiences and add face, hand, and pose tracking to your projects in a snap ✨👌


// Enable Mediapipe's "Hands" model
const handsfree = new Handsfree({hands: true})
// Enable plugins tagged with "browser"
handsfree.enablePlugins('browser')
// Start tracking
handsfree.start()

Run the above code!

  • 👌 Pinch your thumb and index to grab the page
  • ↕ While pinched, move hand up and down to scroll page
Powered by


# Installing from CDN

<head>
  <!-- Include Handsfree.js -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@8.2.2/build/lib/assets/handsfree.css" />
  <script src="https://unpkg.com/handsfree@8.2.2/build/lib/handsfree.js"></script>
</head>

<body>
  <!-- Instantiate and start it -->
  <script>
    const handsfree = new Handsfree({hands: true})
    handsfree.enablePlugins('browser')
    handsfree.start()
  </script>
</body>

# Installing from NPM

# From your projects root
npm i handsfree
// Inside your app
import Handsfree from 'handsfree'

const handsfree = new Handsfree({hands: true})
handsfree.enablePlugins('browser')
handsfree.start()

# Hosting the models yourself

The above will load models, some over 10Mb, from the Unpkg CDN (opens new window). If you'd rather host these yourself (for example, to use offline) then you can eject the models from the npm package into your project's public folder:

# Move the models into your project's public directory
# - change PUBLIC below to where you keep your project's assets

# ON WINDOWS
xcopy /e node_modules\handsfree\build\lib PUBLIC
# EVERYWHERE ELSE
cp -r node_modules/handsfree/build/lib/* PUBLIC
import Handsfree from 'handsfree'

const handsfree = new Handsfree({
  hands: true,
  // Set this to your where you moved the models into
  assetsPath: '/PUBLIC/assets',
})
handsfree.enablePlugins('browser')
handsfree.start()

# Models

MediaPipe Hands

  • 21 2D hand landmarks per hand
  • Track up to 4 hands at once
  • 📅 Extra helpers and plugins coming soon

MediaPipe FaceMesh

  • 468 2D face landmarks
  • Track up to 4 faces at once
  • 📅 Extra helpers and plugins coming soon

MediaPipe Pose

  • Full body mode with 33 2D pose landmarks
  • Upper body mode with 25 2D upper pose landmarks
  • 📅 Extra helpers and plugins coming soon

MediaPipe Holistic

  • 468 2D face landmarks
  • 21 2D hand landmarks per hand with up to 4 hands
  • 33 2D pose landmarks
  • 📅 Extra helpers and plugins coming soon

TensorFlow Handpose

  • 21 3D hand landmarks
  • Can only track 1 hand at a time
  • 📅 Extra helpers and plugins coming soon

Jeeliz Weboji

  • 6DOF head pose estimations
  • 11 face morphs and 16 helper states
  • Comes with "Face Pointer" based plugins

# Quickstart Workflow

The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, and feel free to post on the Google Groups (opens new window) or Discord (opens new window) if you get stuck!

// Let's enable face tracking with the default Face Pointer
const handsfree = new Handsfree({weboji: true})
handsfree.enablePlugins('browser')

// Now let's start things up
handsfree.start()

// Let's create a plugin called "logger"
// - Plugins run on every frame and is how you "plug in" to the main loop
// - "this" context is the plugin itself. In this case, handsfree.plugin.logger
handsfree.use('logger', data => {
  console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this)
})

// Let's switch to hand tracking now. To demonstrate that you can do this live,
// let's create a plugin that switches to hand tracking when both eyebrows go up
handsfree.use('handTrackingSwitcher', ({weboji}) => {
  if (weboji.state.browsUp) {
    // Disable this plugin
    // Same as handsfree.plugin.handTrackingSwitcher.disable()
    this.disable()

    // Turn off face tracking and enable hand tracking
    handsfree.update({
      weboji: false,
      hands: true
    })
  }
})

// You can enable and disable any combination of models and plugins
handsfree.update({
  // Disable weboji which is currently running
  weboji: false,
  // Start the holistic model
  holistic: true,

  // This is also how you configure (or pre-configure) a bunch of plugins at once
  plugin: {
    fingerPointer: {enabled: false},
    faceScroll: {
      vertScroll: {
        scrollSpeed: 0.01
      }
    }
  }
})

// Disable all plugins
handsfree.disablePlugins()
// Enable only the plugins for making music (not actually implemented yet)
handsfree.enablePlugins('music')

// Overwrite our logger to display the original model APIs
handsfree.plugin.logger.onFrame = (data) => {
  console.log(handsfree.model.holistic?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api)
}

Thanks to all of our GitHub Sponsors, including:


Debugger