# Model: Holistic

Overview and basic demo
  • πŸ™‚ 468 2D face landmarks
  • πŸ– 21 2D hand landmarks per hand with up to 4 hands
  • 🀺 33 2D pose landmarks
  • πŸ“… Extra helpers and plugins coming soon

This model doesn't come with any bonuses or plugins yet but they'll come soon. The API will remain exactly the same, so feel free to started with this model today!

☝ Note

While this model uses a collection of models similar to MediaPipe Hands, Pose, and FaceMesh, they only contain the keypoints and not the other models data points. Plugins for those specific models aren't working for this Holistic model yet, but they will be soon.

# Usage

# With defaults

const handsfree = new Handsfree({holistic: true})
handsfree.start()

# With config

const handsfree = new Handsfree({
  holistic: {
    enabled: false,
    
    // Outputs only the top 25 pose landmarks if true,
    // otherwise shows all 33 full body pose landmarks
    // - Note: Setting this to true may result in better accuracy 
    upperBodyOnly: true,

    // Helps reduce jitter over multiple frames if true
    smoothLandmarks: true,

    // Minimum confidence [0 - 1] for a person detection to be considered detected
    minDetectionConfidence: 0.5,
        
    // Minimum confidence [0 - 1] for the pose tracker to be considered detected
    // Higher values are more robust at the expense of higher latency
    minTrackingConfidence: 0.5
  }
})

handsfree.start()

# Data

# Left Hand Landmarks


Image source, MediaPipe: https://google.github.io/mediapipe/solutions/hands#hand-landmark-model (opens new window)

// An array of landmark points for the left hand
handsfree.data.holistic.leftHandLandmarks == [
  // Landmark 0
  {x, y},
  // Landmark 1
  {x, y},
  // ...
  // Landmark 20
  {x, y}
]

// landmark 0
handsfree.data.holistic.leftHandLandmarks[0].x
handsfree.data.holistic.leftHandLandmarks[0].y

# Right Hand Landmarks

// An array of landmark points for the right hand
handsfree.data.holistic.rightHandLandmarks == [
  // Landmark 0
  {x, y},
  // Landmark 1
  {x, y},
  // ...
  // Landmark 20
  {x, y}
]

// landmark 0
handsfree.data.holistic.rightHandLandmarks[0].x
handsfree.data.holistic.rightHandLandmarks[0].y

# Face Landmarks

// An array of landmark points for the face
handsfree.data.holistic.faceLandmarks == [
  // Landmark 0
  {x, y},
  // Landmark 1
  {x, y},
  // ...
  // Landmark 467
  {x, y}
]

// landmark 0
handsfree.data.holistic.faceLandmarks[0].x
handsfree.data.holistic.faceLandmarks[0].y

# Pose Landmarks


Image source, MediaPipe: https://google.github.io/mediapipe/solutions/pose#pose-landmark-model-blazepose-tracker (opens new window)

// An array of landmark points for the face
handsfree.data.holistic.poseLandmarks == [
  // Landmark 0
  {x, y, visibility},
  // Landmark 1
  {x, y, visibility},
  // ...
  // Landmark 32
  {x, y, visibility}
]

// landmark 0
handsfree.data.holistic.poseLandmarks[0].x
handsfree.data.holistic.poseLandmarks[0].y
// The confidence in this pose landmark
handsfree.data.holistic.poseLandmarks[0].visibility

# Examples of accessing the data

handsfree = new Handsfree({holistic: true})
handsfree.start()

// From anywhere
handsfree.data.holistic.multiFaceLandmarks

// From inside a plugin
handsfree.use('logger', data => {
  if (!data.holistic?.poseLandmarks) return

  console.log(data.holistic.poseLandmarks[0])
})

// From an event
document.addEventListener('handsfree-data', event => {
  const data = event.detail
  if (!data.holistic?.leftHandLandmarks) return

  console.log(data.holistic.leftHandLandmarks)
})

# Examples

Flappy Pose
Person playing Flappy Bird by flapping their arms. Flappy Bird is a game where you must flap the birds wings to fly or dodge barriers

In this game, the goal is to flap your arms to get the bird to fly around dodging obstacles. Made with an older version of Handsfree, but the API is very similar!

Add your project
If you've made something with this model I'd love to showcase it here! DM me on Twitter, make a pull request, or find us on Discord.
Debugger