Pandata Blog

AI design and development for high risk industries

More and more of our clients approach us with operational problems that are ripe for machine learning solutions.  As such, the data scientists at Pandata spend much of our time maintaining proficiency with the latest results coming out of the machine learning research community, and I tend to focus on research involving  “differentiable programming”, otherwise known as deep learning.  This research community coalesces at the Neural Information Processing Systems conference every year, so I decided to attend this past December.

Not all of these developments are relevant to commercial applications, and even some of the relevant ones aren’t yet ready to be developed into reliable products or processes.  But several topics were covered that are critical to any company considering using machine learning to accelerate their core operations.

Firstly, and most importantly, there’s an issue of social bias in machine learning.  Anyone who works with these algorithms regularly knows that a machine learning model is only as good as the data it’s provided — “garbage in, garbage out,” as they say.  It turns out that much of our data is as biased as we are when it comes to controversial topics — one only need think of Microsoft’s chatbot Tay or this clip on bias in Google translate from Andrew Plassard’s recent talk at the Cleveland AI Group.  At NIPS, Kate Crawford gave an excellent talk on this topic.  Special care must be taken to ensure that our algorithms are not unfairly impacting certain groups, and companies implementing machine learning without proper expertise are exposing themselves to significant operational and reputational risk.

Secondly, interest in security seems to be at an all-time high.  Pandata has covered similar topics but that had more to do with using machine learning to improve cybersecurity defenses.  In contrast, much of the interest at this year’s NIPS was around the security of machine learning systems themselves.  It turns out that machine learning algorithms can be sensitive to attacks just like other software, and the vulnerabilities are even harder to detect and correct!  This is vital for systems that are designed to ingest data generated directly from users and sensors — some sensitive examples include check deposit systems for banking apps, facial recognition software on your phone, and self-driving cars.  Some attacks can be prevented using traditional cybersecurity processes and controls, while others cannot.  In order to protect against these kinds of vulnerabilities in user-facing systems, it’s best to use special techniques during model development, consulting with an expert on adversarial defense testing if needed.

With all these risks, why would anyone want to build user-facing products and processes with machine learning?  This is, of course, because the community has been able to accomplish some amazing things with this technology.  This was particularly apparent during the Machine Learning for Healthcare workshop at NIPS.  Dr. Fei Fei Li spoke about her work with the Partnership for AI-Assisted Care, where they’re improving hand-washing compliance, ICU patient monitoring, and senior wellbeing with computer vision.  Jure Leskovec also presented on an exciting project attempting to identify FDA-approved drugs that can be repurposed for more effective therapies of diseases they weren’t developed to treat.  AI has yet to fully diffuse into the healthcare industry, which means this kind of care will likely be a key competitive differentiator as the industry attempts to improve patient satisfaction while maintaining current revenue expectations.