OtoSense: the next level in sound-based IoT

It sounds (pardon the pun) as if the IoT may really be taking off as an important diagnostic repair tool.

I wrote a while ago about the Auguscope, which represents a great way to begin an incremental approach to the IoT because it’s a hand-held device to monitor equipment’s sounds and diagnose possible problems based on abnormalities.

Now NPR reports on a local (Cambridge) firm, OtoSense, that is expanding on this concept on the software end. Its tagline is “First software platform turning real-time machine sounds and vibrations into actionable meaning at the edge.”

Love the platform’s origins: it grows out of founder Sebastien Christian’s research on deafness (as I wrote in my earlier post, I view suddenly being able to interpret things’ sounds as a variation on how the IoT eliminates the “Collective Blindness”  that I’ve used to describe our past inability to monitor things before the IoT’s advent):

“[Christian} … is a quantum physicist and neuroscientist who spent much of his career studying deaf children. He modeled how human hearing works. And then he realized, hey, I could use this model to help other deaf things, like, say, almost all machines.”

(aside: I see this as another important application of my favorite IoT question: learning to automatically ask “who else can use this data?” How does that apply to YOUR work? But I digress).

According to Technology Review, the company is concentrating primarily on analyzing car sounds from IoT detectors on the vehicle at this point (working with a number of car manufacturers) although they believe the concept can be applied to a wide range of sound-emitting machinery:

“… OtoSense is working with major automakers on software that could give cars their own sense of hearing to diagnose themselves before any problem gets too expensive. The technology could also help human-driven and automated vehicles stay safe, for example by listening for emergency sirens or sounds indicating road surface quality.

OtoSense has developed machine-learning software that can be trained to identify specific noises, including subtle changes in an engine or a vehicle’s brakes. French automaker PSA Group, owner of brands including Citroen and Peugeot, is testing a version of the software trained using thousands of sounds from its different vehicle models.

Under a project dubbed AudioHound, OtoSense has developed a prototype tablet app that a technician or even car owner could use to record audio for automated diagnosis, says Guillaume Catusseau, who works on vehicle noise in PSA’s R&D department.”

According to NPR, the company is working to apply the same approach to a wide range of other types of machines, from assembly lines to DIY drills. As always with IoT data, handling massive amounts of data will be a challenge, so they will emphasize edge processing.

OtoSense has a “design factory” on the site, where potential customers answer a variety of questions about the sounds they must monitor (such as whether the software will be used indoors or out, whether it is to detect anomalies, etc. that will allow the company to choose the appropriate version of the program.

TechCrunch did a great article on the concept, which underscores really making sound detection precise will take a lot of time and refinement, in part because of the fact that — guess what — sounds from a variety of sources are often mingled, so the relevant ones must be determined and isolated:

“We have loads of audio data, but lack critical labels. In the case of deep learning models, ‘black box’ problems make it hard to determine why an acoustical anomaly was flagged in the first place. We are still working the kinks out of real-time machine learning at the edge. And sounds often come packaged with more noise than signal, limiting the features that can be extracted from audio data.”

In part, as with other forms of pattern recognition such as voice, this is because it will require accumulating huge data files:

“Behind many of the greatest breakthroughs in machine learning lies a painstakingly assembled dataset.ImageNet for object recognition and things like the Linguistic Data Consortium and GOOG-411 in the case of speech recognition. But finding an adequate dataset to juxtapose the sound of a car-door shutting and a bedroom-door shutting is quite challenging.

“’Deep learning can do a lot if you build the model correctly, you just need a lot of machine data,’ says Scott Stephenson, CEO of Deepgram, a startup helping companies search through their audio data. ‘Speech recognition 15 years ago wasn’t that great without datasets.’

“Crowdsourced labeling of dogs and cats on Amazon Mechanical Turk is one thing. Collecting 100,000 sounds of ball bearings and labeling the loose ones is something entirely different.

“And while these problems plague even single-purpose acoustical classifiers, the holy grail of the space is a generalizable tool for identifying all sounds, not simply building a model to differentiate the sounds of those doors.

…”A lack of source separation can further complicate matters. This is one that even humans struggle with. If you’ve ever tried to pick out a single table conversation at a loud restaurant, you have an appreciation for how difficult it can be to make sense of overlapping sounds.

Bottom line: there’s still a lot of theoretical and product-specific testing that must be done before IoT-based sound detection will be an infallible diagnostic tool for predictive maintenance, but clearly there’s precedent for the concept, and the potential payoff are great!


LOL: as the NPR story pointed out, this science may owe its origins to two MIT grads of an earlier era, “Click” and “Clack” of Car Talk, who frequently got listeners to contribute their own hilarious descriptions of the sounds they heard from their malfunctioning cars.   BRTTTTphssssBRTTTT…..

Sound’s emerging IoT role

Could sound be a critical IoT tool?

I’d fixated in the past on a metaphor I called “Collective Blindness,” as a way to explain how difficult it used to be to get accurate, real-time data about how a whole range of things, from tractors to your body, were actually working (or not) because we had no way to penetrate the surface of these objects as they were used. As a result, we created some not-so-great work-arounds to cope with this lack of information.

Then along came the IoT, and no more collective blindness!

Now I’m belatedly learning about some exciting efforts to use another sense, sound, for the IoT.  Most prominent, of course, is Amazon’s Alexa and her buddies (BTW, when I ask Siri if she knows Alexa, her response was an elusive “this is about you, not me,” LOL), but I’ve found a variety of start-ups pursuing quite different aspects of sound. They nicely illustrate the variety of ways sound might be used.

technician using Auguscope to detect   sound irregularities in machinery

First is Augury.

What I particularly love about their device and accompanying smartphone app it is that they are just about the lowest-cost, easiest-to-use, rapid payback industrial IoT devices I can think of.

That makes them a great choice to begin an incremental approach to the IoT, testing the waters by some measures that can be implemented quickly, pay rapid bottom-line benefits and therefore may lure skeptical senior management who might then be willing to then try bolder measures   (this incremental approach was what I outlined in my Managing the Internet of Things Revolution e-guide for SAP, and I’ll be doing a webinar on the approach in April with Mendix, which makes a nifty no-code, low-code tool).

Instead of requiring built-in sensors, an Auguscope is a hand-held device that plant personnel can carry anywhere in the building it’s needed to analyze how the HVAC system is working. A magnetic sensor temporarily attaches to the machine and the data flows from the Auguscope to the cloud where it is analyzed to see if the sound is deviating from pre-recorded normal sounds, indicating maintenance is needed. Consistent with other IoT products that are marketed as services instead of sold, it uses a “Diagnostics as a Service” model, so there are no up-front costs and customers pay as they go. The company hopes that the technology will eventually be built into household appliances such as washers and dryers.

Presenso is the second company using sound to enable predictive maintenance.  It is sophisticated cloud-based software that takes data from a wide range of already-installed sensors and interprets any kind of data: sound, temperature, voltage, etc.  It builds a model of the machine’s normal operating data and then creates visualizations when the data varies from the norm. Presenso’s power comes from combining artificial intelligence and big data.

Finally, and most creative is Chirp (hmm: Chrome wouldn’t let me enter their site, which it said was insecure. Here’s the URL:www.chirp.io/ — try at your own risk…) , a UK company that transmits data using audio clips that really sound like chirps. It’s amazing!  Check out this video of an app in India that uses sound to pay fares on the country’s version of Uber:

Another Chirp app is a godsend to all who forget Wi-Fi passwords: your phone “chirps” a secure access code, allowing you to join the network automatically.   The company has released iOS and Android versions.  As VentureBeat reported:

“Each chirp lasts a couple of seconds, and the receiving device “listens” for a handful of notes played quickly in a certain order, in a certain range, and at a certain speed. While there are other easy ways of sharing files and data in real-time, such as Bluetooth, Chirp doesn’t require devices to pair in advance, there is no need to set up an account, and it’s ultimately a much quicker way of sharing files.

“That said, with Chirp, the file itself isn’t sent peer-to-peer, and the data doesn’t actually travel directly via audio. Chirp merely decodes and encodes the file, with the associated sound serving as the delivery mechanism. A link is generated for the recipient(s) to access it on Chirp’s servers, but the process from sending to receiving is seamless and near-instant.”

In terms of IoT applications, it could also connect with physical objects (hmm: retailing uses??). The Chirp platform is so cool that I suspect it will be a global hit (the company says it’s already used in 90 countries).

So, I’ve had my senses opened: from now on, I’ll add voice and sound in general to the list of cool IoT attributes.  Because voice and sound are so ubiquitous, they really meet the late Mark Weiser’s test:  “the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” What could be more woven into the fabric of everyday life than sound — and, potentially, more valuable?

BTW: let me put in a plug for another IoT voice product. During the past two months, I recorded 7 hours of my voice speaking a very strange mishmash of sentences drawn from, among others, Little Women, Jack London’s Call of the Wild, The Wizard of Oz, and The Velveteen Rabbit (I worried about the she-wolf sneaking up on Meg, LOL….). Using the algorithms developed for Alexa, the Vocal ID team will slice and dice my voice and create a natural sounding one for someone who cannot speak due to a birth defect or disease.  I hope you’ll join me in volunteering for this wonderful program.