How Neurobit Saves 60% Of Their Cloud Costs and Serves Machine Learning at Scale

Neurobit is leading the way in Health Tech AI research

Neurobit is a digital health company based in New York, Singapore, and Bangalore. They are developing technologies to predict and plan for adverse health outcomes well before they happen, using vitals collected during sleep as a biomarker.

The company has constructed the world's largest sleep database with over a trillion data points of multi-channel physiological data. The scale of data that they have trained their models gives them the robustness to generalize the model to any new scenario. Their use case resembles many recent AI efforts that new age Health Tech companies and new AI initiatives of tech majors.

We found similarities between Neurobit’s use cases and those of other enterprise and mid-size healthcare companies that we have talked to:

1. Highly regulated with the utmost strict guidelines around data privacy
2. Large dataset size
3. Low fault tolerance to failed model requests
4. Use of proprietary algorithms with strong IP safeguarding regulations

The team was losing $1000+ for each user that they failed to serve

When we first met the Neurobit team, they had been conducting medical trials with 120+ research centers, universities, and 1000+ subjects. Most of these subjects were in the same geographical location.

When the person wakes up, the sensors send physiological data to the server for processing. Each request needs 20 different models to be called to generate the final output.

The data size that came in with each request was large (400 mb+) and during the high traffic time, the team could see a significant lag in the response time and even the dropping of requests with loss of user data.

‍

This situation was causing significant adverse financial implications for the team:

1. Dropping a singular request coming in from the trial subjects cost the company north of $1000 in experiment charges.
2. The company could pace up the medical trials by working with labs and hospitals which upload data in large batches delaying the product approvals.

As in other machine learning use cases within the health tech industry, the team could not afford the loss of customer data or delayed responses.

The team wanted to deploy their models on Kubernetes

The team knew that deploying their models on Kubernetes, with a queue to store requests before they are processed, would be able to solve their reliability issues.

However, the IP protection norms of the company limited access to the model only to a few members of the Machine Learning team and not to the DevOps team.

The Machine Learning team had limited bandwidth and expertise in Kubernetes to pull this off themselves. Instead, they wanted to work on developing new models.

The existing stack that was being used for machine learning deployments was:

‍

The stack worked fine for the team up to a certain scale. But once the use case scaled, the team started facing reliability issues with serving the model that needed immediate attention.

The team wanted to strengthen the security of its model APIs

Since the company deals with sensitive PII and health data, maintaining the security of the model APIs was of utmost importance to them. They wanted no customer data to be leaving their cloud and to strengthen the authentication and security norms of the APIs that they had been using.

The team decided to partner with TrueFoundry

The team needed a way to empower their Machine Learning team, which had access to the model, to be able to deploy and manage models on Kubernetes independently. The objectives that the Neurobit team wanted to achieve through a partnership with TrueFoundry were:

1. To increase the reliability of their ML models when they are serving at scale
2. To empower the DS team to be able to deploy and manage the machine learning models on Kubernetes
3. To strengthen the authentication and security protocols on the model APIs

TrueFoundry helped the team solve for reliability and security

The platform was setup up in less than a day

The TrueFoundry team helped the Neurobit team could install the TrueFoundry agent and control plane on their cluster within a 2-hour call. They were informed of the access and permissions needed, and they were walked through each step of the installation in a single call.

The team was given a choice to install only the modules of the TrueFoundry platform that were relevant to them (Model deployment and authentication).

Post the installation, the team was given a demo of the platform and handed over the documentation.

The Neurobit team started deploying from Day 1

The Neurobit team was able to start using the platform for their model deployments right from day 1. They could directly connect their Git repositories to the platform, this code was automatically dockerized and deployed on the platform by using the TrueFoundry UI, APIs, or the Python SDK. There were no code changes required and no need to learn any additional framework for all the workflows that the team was trying to complete.

The team showed great pace since they wanted to solve the reliability issues fast. Within a few days, they started exploring more and more features of the platform and provided us with feedback.

Within two weeks, the team was able to:

1. Shift their ML workloads entirely on Kubernetes with TrueFoundry.
2. Deploy ML model with async queue to store the incoming requests when burst traffic is detected, configured through a simple flag.
3. Optimize the resource allocation for the ML services as per the traffic patterns and scale down resources provisioned to save costs when request volumes are low.
4. Set up authentication and harden the security on all the model API endpoints.

Impact on Machine Learning Use Cases

Through the deployment of Machine Learning models on TrueFoundry, the team was able to:

1. Save ~$25-30K in terms of trial subjects payout by bringing the model failure and reliability issues to 0.
2. 35-40 % reduction in cloud costs
3. 3-6 month faster clinical trials through collaboration with hospitals and labs
4. Set up authentication on the model API endpoint

TrueFoundry Team Also Helped the Neurobit Team Optimize Their Software Architecture

When the highest priority challenges with the Machine Learning models was solved, the TrueFoundry team decided to go further to make sure that the Neurobit team was set up for success. During our conversations with the Neurobit team, we got to understand that there was scope for the microservices architecture of the company to be optimized further. This could have had a possible impact on the inference time and cloud costs that the team was incurring.

We ended up doing an in-depth review of the microservices architecture with the team.

We got the following understanding of the architecture that the team was following:

1. There were 5 different microservices that were being invoked for processing the input that was received from the users.
2. Each service was writing its intermediate output to a database. From this database, the next service was reading output of the previous service, running the computation on it, and writing it back to the database for the next microservice to consume.

This entire process took ~7 minutes for each request.

Optimized Microservices Architecture with TrueFoundry

‍

We tried to understand the fault tolerance and inference times that the team required. With this understanding, we suggested the Neurobit team directly pass the output of one service to the other over the gRPC protocol.

The advantage of this architecture was that.

1. Since each microservice took ~30 seconds to execute, there was not much risk of losing intermediate outputs in case there was a failure since the entire pipeline could be run again.
2. It significantly reduces the cost of data transfer and the time taken to write the intermediate outputs to a Database.

This new pipeline was hosted on the TrueFoundry platform and it decreased the model inference time from ~7 mins/request to ~2 mins/request.

Impact Of The Architecture Redesign

1. The inference time of the ML pipeline was reduced from ~7 minutes to ~2 minutes
2. The cost of running the Machine Learning service was reduced by 60-70%
3. Increase in developer productivity and independence. Developers could now deploy models and applications on their own which earlier needed back and forths with the DevOps team
4. The team is now running all of their models and applications fully on Kubernetes

Impact on Business

As our partnership with the Neurobit team progresses, we have seen the business realise the benefits from the faster response times, reliability, and scale that the TrueFoundry platform has helped the Neurobit team achieve.

‍

Impact on Technical Stack of Neurobit

TrueFoundry helped the Neurobit team to move all of their Machine Learning workloads to Kubernetes without having to deal with the complexity of learning anything new related to Kubernetes. It has also helped the team become independent in handling all the advanced operations in Kubernetes like doing Async deployments, setting up autoscaling, serverless deployments, etc.

We have also been able to help the team move some of their software resources onto microservices architecture on top of Kubernetes so that their stack is future-proof and running with optimum utilization levels.

‍

Our Learnings From The Engagement With Neurobit

As we keep on engaging with Neurobit and helping them achieve the scale and level of impact with Artificial Intelligence that they have set out to, we are grateful for all the learnings that we have been able to derive from engaging with the team. It has helped shape both, how we think about engagement with clients as well as given a solid direction to our product.

Some of our core learnings include:

1. Companies can save a major chunk (>40%)of their cloud costs by using their resources optimally
2. Making the developers independent and empowering them to do the releases themselves, increases the pace at which the team can ship
3. Starting out with a scale-ready stack ensures that things do not break and that the team does not end up making the extra effort of migration eventually

We have co-developed some important features of the platform while trying to solve the use cases that the Neurobit team required us to enable. These include:

1. Async Deployments
2. Hosted Jupyter Notebooks

Way Forward

We look forward to engaging with the Neurobit team in the long term and getting to learn from them while trying to help them along the way. Some of the future developments that could be in store for this engagement include:

1. Scaling the model inference throughputs to 10X the current scale
2. Helping to migrate the entire software stack of Neurobit on TrueFoundry
3. Deployment of new experimental models and running pilots on them with labs, hospitals, and nursing homes.

Excited to see what comes next!