OtoSense: the next level in sound-based IoT

It sounds (pardon the pun) as if the IoT may really be taking off as an important diagnostic repair tool.

I wrote a while ago about the Auguscope, which represents a great way to begin an incremental approach to the IoT because it’s a hand-held device to monitor equipment’s sounds and diagnose possible problems based on abnormalities.

Now NPR reports on a local (Cambridge) firm, OtoSense, that is expanding on this concept on the software end. Its tagline is “First software platform turning real-time machine sounds and vibrations into actionable meaning at the edge.”

Love the platform’s origins: it grows out of founder Sebastien Christian’s research on deafness (as I wrote in my earlier post, I view suddenly being able to interpret things’ sounds as a variation on how the IoT eliminates the “Collective Blindness”  that I’ve used to describe our past inability to monitor things before the IoT’s advent):

“[Christian} … is a quantum physicist and neuroscientist who spent much of his career studying deaf children. He modeled how human hearing works. And then he realized, hey, I could use this model to help other deaf things, like, say, almost all machines.”

(aside: I see this as another important application of my favorite IoT question: learning to automatically ask “who else can use this data?” How does that apply to YOUR work? But I digress).

According to Technology Review, the company is concentrating primarily on analyzing car sounds from IoT detectors on the vehicle at this point (working with a number of car manufacturers) although they believe the concept can be applied to a wide range of sound-emitting machinery:

“… OtoSense is working with major automakers on software that could give cars their own sense of hearing to diagnose themselves before any problem gets too expensive. The technology could also help human-driven and automated vehicles stay safe, for example by listening for emergency sirens or sounds indicating road surface quality.

OtoSense has developed machine-learning software that can be trained to identify specific noises, including subtle changes in an engine or a vehicle’s brakes. French automaker PSA Group, owner of brands including Citroen and Peugeot, is testing a version of the software trained using thousands of sounds from its different vehicle models.

Under a project dubbed AudioHound, OtoSense has developed a prototype tablet app that a technician or even car owner could use to record audio for automated diagnosis, says Guillaume Catusseau, who works on vehicle noise in PSA’s R&D department.”

According to NPR, the company is working to apply the same approach to a wide range of other types of machines, from assembly lines to DIY drills. As always with IoT data, handling massive amounts of data will be a challenge, so they will emphasize edge processing.

OtoSense has a “design factory” on the site, where potential customers answer a variety of questions about the sounds they must monitor (such as whether the software will be used indoors or out, whether it is to detect anomalies, etc. that will allow the company to choose the appropriate version of the program.

TechCrunch did a great article on the concept, which underscores really making sound detection precise will take a lot of time and refinement, in part because of the fact that — guess what — sounds from a variety of sources are often mingled, so the relevant ones must be determined and isolated:

“We have loads of audio data, but lack critical labels. In the case of deep learning models, ‘black box’ problems make it hard to determine why an acoustical anomaly was flagged in the first place. We are still working the kinks out of real-time machine learning at the edge. And sounds often come packaged with more noise than signal, limiting the features that can be extracted from audio data.”

In part, as with other forms of pattern recognition such as voice, this is because it will require accumulating huge data files:

“Behind many of the greatest breakthroughs in machine learning lies a painstakingly assembled dataset.ImageNet for object recognition and things like the Linguistic Data Consortium and GOOG-411 in the case of speech recognition. But finding an adequate dataset to juxtapose the sound of a car-door shutting and a bedroom-door shutting is quite challenging.

“’Deep learning can do a lot if you build the model correctly, you just need a lot of machine data,’ says Scott Stephenson, CEO of Deepgram, a startup helping companies search through their audio data. ‘Speech recognition 15 years ago wasn’t that great without datasets.’

“Crowdsourced labeling of dogs and cats on Amazon Mechanical Turk is one thing. Collecting 100,000 sounds of ball bearings and labeling the loose ones is something entirely different.

“And while these problems plague even single-purpose acoustical classifiers, the holy grail of the space is a generalizable tool for identifying all sounds, not simply building a model to differentiate the sounds of those doors.

…”A lack of source separation can further complicate matters. This is one that even humans struggle with. If you’ve ever tried to pick out a single table conversation at a loud restaurant, you have an appreciation for how difficult it can be to make sense of overlapping sounds.

Bottom line: there’s still a lot of theoretical and product-specific testing that must be done before IoT-based sound detection will be an infallible diagnostic tool for predictive maintenance, but clearly there’s precedent for the concept, and the potential payoff are great!


LOL: as the NPR story pointed out, this science may owe its origins to two MIT grads of an earlier era, “Click” and “Clack” of Car Talk, who frequently got listeners to contribute their own hilarious descriptions of the sounds they heard from their malfunctioning cars.   BRTTTTphssssBRTTTT…..

Testing the IoT Waters: 1st Steps in Creating an IoT Corporate Strategy

What if you’re interested in the Internet of Things, but are a little scared of making a major commitment and making major expenditures until you build your familiarity level and start to enjoy some tangible results?

That concern is understandable, especially when prognosticators such as I emphasize what a transformational impact the IoT will have on every aspect of your operations and strategy.

So where to begin?

I’ll speak on this issue at SAP’s  IoT 2016 Conference, Feb. 16-19, in Las Vegas, and hope you can attend. But, if not, or if a teaser might convince you to make the plunge, here’s a summary of my major points, which I hope will motivate you to act sooner, rather than later!

Managing_the_Internet_of_Things_RevolutionThis is an issue that I first visited with my “Managing the Internet of Things Revolution” e-guide to IoT strategy for C-level executives, which I wrote in 2014 for SAP, and which has been successful enough that they’ve translated it into eight languages.

I suggested that the best reason to begin now on creating and executing an IoT strategy was that a lot of the requisite tools for an IoT strategy were also critical to optimize your current operations:

  • invest now in analytical tools (such as SAP’s HANA!), so that you can make sense of the rapidly-expanding amount of data (especially unstructured data) that you are already collecting, with new benefits including predictive analytics that allow you to better predict the future.
  • even before capital equipment is redesigned to incorporate sensors that will yield 24/7 real-time data on their operations and status, consider add-on sensors where available, so you can take the guesswork out of operations.
  • where possible, process sensor data “at the edge,” so that only the relevant data will be conveyed to your processing hub, reducing storage and central processing demands.
  • develop or contract for cloud storage, to handle vastly increased data.
GE Brilliant Factory benefits

GE Brilliant Factory benefits

As I’ll explain my speech, even without launching any major IoT projects such as product redesign or converting products into services, initial IoT projects such as these will dramatically boost your profits and efficiency by allowing unprecedented precision in operations.  I’ll emphasize the example of GE, whose “Brilliant Factory” initiative is aimed at increasing both its own manufacturing efficiency and its customers’ as well. They make a modest, but astonishing claim:

“GE estimates that a 1% improvement in its productivity across its global manufacturing base translates to $500 million in annual savings. Worldwide, GE thinks a 1% improvement in industrial productivity could add $10 trillion to $15 trillion to worldwide GDP over the next 15 years.”

Remember: that’s not exploiting the full potential of the IoT, but simply using it to boost operating efficiency. I see this as bringing about an era of “Precision Manufacturing,” because everyone who needs real-time data about the assembly line and production machinery will be able to share it instantly — including not only all departments within your company but also your supply chain and your distribution network.

In many cases, resupply will be automatic, through M2M processes where data from the assembly line will automatically trigger supply re-orders (and may lead to reshoring of jobs, because the advantages of true “just-in-time” delivery of parts from a supplier located a few miles away will outweigh the benefits of using one on the other side of the world, where delivery times are measured in weeks).  Instead of the current linear progression from supply chain to factory floor to distribution network, we’ll have a continuous loop uniting all of those components, with real-time IoT data as the “hub.”

Again, without making a full-fledged commitment to the IoT, another benefit that I’ll detail is how you’ll be able to dramatically improve workplace safety, especially inherently chaotic and fast-changing worksites such as construction projects and harbors, whose common elements include unpredictable schedules, many companies and contractors, many workers, and many vehicles — a recipe for disaster given current conditions!  However, the combination of simply putting location sensors on the equipment, vehicle, and people can radically decrease the risk. For example,  in Dubai — home to 25% of all construction cranes in the world — SAP partnered with a worldwide leader in construction site safety, SK Solutions. Sensors are located on machinery throughout every site, reporting real-time details about every activity: machinery’s position, movement, weight, and inertia and critical data from other sources (as with the GE Durathon factory’s use of weather data), including wind speed and direction, temperature, and more. Managers can detect potential collisions, and an auto-pilot makes instant adjustments to eliminate operator errors. “The information is delivered on dashboards and mobile devices, visualized with live 3-D images with customizable views.”

As I’ll tell the conference attendees,

“Equally incredible is the change at the Port of Hamburg, Germany’s biggest port, which must juggle 9 million containers and 12,000 vessels a year, not to mention a huge number of trucks and trains. You can imagine the potential for snarls and accidents. Since installing HANA, all of these components, including the drivers and other operators, are linked in real time.  Average waiting time for each truckload has been cut 5 minutes,  and there are 5,000 fewer truck hours daily. The coordination has gotten so precise that, if a trucker will be held up by a bridge opening, the nearby coffee shop will send a discount coupon to his iPad.”

I’ll conclude by mentioning a couple of the long-term components of an IoT strategy, such as redesigning products so that they can be controlled by apps and/or feedback constant information on their status, and considering whether to market products instead as services, where the customer only pays for the products when they’re actually being used, and creating optional data services that customers may choose to buy because they’ll allow the customer to optimize operating efficiency.

But the latter are the long-term challenges and benefits.  For now, I’ll tell the audience that the important thing is to begin now investing in the analytical tools and sensors that will help them boost efficiency.

Hope you can be there!

Oh yeah. Why get started on your IoT strategy now, rather than wait a few more years? Last year, former Cisco Chairman John Chambers said that 40% of the companies attending a recent seminar wouldn’t survive in a “meaningful way” within 10 years if they don’t begin now to embrace the IoT. Sobering, huh?

My speech on how the Internet of Things will aid Predictive Analytics

I spoke yesterday at the Predictive Analytics Manufacturing conference in Chicago, about a theme I first raised in the O’Reilly SOLID blog, about how the Internet of Things could bring about an “era of precision manufacturing.”

I argued that, as powerful as Predictive Analytics tools have been in analyzing manufacturing data and improving forecasting, their effectiveness has been artificially restricted because, for example, we can’t “see” inside production machinery to detect early signs of metal fatigue in time to avoid a costly breakdown, nor can we tell whether EVERY product on an assembly line will function when customers use them.

By contrast, I argued that the IoT will give us all this information, and, most important, allow everyone (from your supply chain and distribution network to EVERYONE in your company) to share this data on a real-time basis.  I warned that it will be management issues (those pesky IoT Essential Truths again!), such as whether to allow this sharing to take place, and whether to end departmental silos, that will be the biggest potential barrier to full IoT implementation.

Believe me, it will be an incredible transformation.  You can read the full text here.