news

Meet the ESP-EYE-Powered Doc InSight

Reporting from Shanghai, China
Aug 27, 2021

Doc InSight provides an augmented reality heads-up display driven by ESP-EYE, and built into a pair of glasses.

Doc InSight is a prototype created by Nick Bild, who is a professional software engineer and maker with over 15 years of experience. Nick’s latest project is an ESP32-driven augmented reality heads-up display, built into a pair of glasses, for healthcare professionals. Relevant medical information from an electronic health record is displayed in the wearer’s field of vision when certain QR codes are detected. The information display is completely passive and hands-free. No action is required of the user.

A machine learning (ML) algorithm runs in the background to enhance patient safety. In particular, two of the most common medical errors are treatment-related infections and medication errors. Doc InSight seeks to eliminate these types of errors without saddling healthcare providers with additional procedures that may either reduce their time treating patients, or lead to physician burnout.

More specifically, an ESP-EYE development board with an OV2640 image sensor captures images of what the wearer of Doc InSight sees, and runs two primary algorithms against those images: a TensorFlow Lite for Microcontrollers’ neural network, and a decoding algorithm for QR codes.

Two types of QR codes are currently detected: those affixed to patients, revealing their name, age, gender, etc., and those affixed to medications, revealing whether they have been previously prescribed to the patient under examination, or whether they may cause any allergies or other unwanted side-effects. All this information is displayed on a transparent LCD within the wearer’s field of vision.

Additionally, a custom convolutional neural network was developed and trained in TensorFlow Lite for hand-washing recognition. The model was quantized and converted into an appropriate structure for use with a resource-constrained microcontroller. Thus, for example, if the provider has not washed their hands (as determined by the ML algorithm) within two minutes of encountering the patient, a hand-washing reminder is also displayed.

Bill of Materials

    • Espressif’s ESP-EYE development board with OV2640 image sensor
    • Extended length flat ribbon cable for the image sensor
    • Nokia 5110 LCD
    • Adafruit QT Py
    • Adafruit Powerboost 500 Charger
    • 2000 mAh LiPo battery
    • Micro USB breakout board
    • USB-C breakout board
    • Miscellaneous wires
    • Wooden dowels
    • Hot glue
    • Glasses frame
    • Metal disc or similar for counterweight

To get all the details and code of Nick Bild’s project, you can check out the post he authored on hackster.io.

Share this article
Reuse this content