Games 24x7 Personalizing Gaming with AI for its 100 Million Users

Improving gaming with AI for 100 Million+ gamers with Games 24x7

Games 24x7 is one of the pioneering gaming companies from Southeast Asia. Headquartered in Mumbai, the company has an active user base of more than 100 Mn gamers. The company has a portfolio of multiple skill and casual games and is one of the leaders in Real Money Gaming. Their two most popular games are RummyCircle (5 Billion Games played in 2023) and My11Circle (More than 500 Million teams made in 2023).

One of the chief differentiators of the company is its ability to use Artificial Intelligence (AI) and behavioral science to make the game-playing experience more immersive. This is facilitated by innovation done by the company’s Artificial Intelligence and Data Science team. When we started interacting with the team they wanted to supercharge the speed of shipping AI projects by removing a few blockers that they felt were holding them back. These included:

1. Delays due to back and forth with the engineering team: The data science team was dependent on their infrastructure-related needs on the engineering team. This back-and-forth for testing and shipping models cost the ML team a lot of delay.
2. High cloud cost and lack of autoscaling: The team was habituated to observe the service during the load testing and provisioned a sufficiently large instance to accommodate the traffic. They were overspending on cloud costs even during low traffic periods due to a lack of autoscaling.
3. Absence of a common deployment method: The data scientists often wasted time working on components related to Kubernetes, Infrastructure, and deployment which was not their strong suit. They also did not have a common deployment method and Data Scientists had to carry out such actions every time they wanted to do a deployment.

The TrueFoundry team partnered with the team to solve these problems. Using the TrueFoundry platform, the team was able to:

1. Reduce the time of deployment of new models and projects by >70%
2. Create a common deployment method companywide, with best practices like git integration, secret management, autoscaling, etc. built in.
3. Create more visibility and ease of control for the engineering team

Games 24x7 is revolutionising gaming with Artificial Intelligence

Games24x7 is a scientific gaming company that specializes in using behavioral science, technology, and artificial intelligence to provide awesome gaming experiences across its platforms.

The company operates in the dynamic and evolving landscape of the online gaming industry, which has witnessed significant growth in India and globally. Online gaming has become a major entertainment segment, attracting a diverse audience. The company aims to capture a majority of the market share in gaming, especially in India which boasts of more than 550 Million players.

Some of its most popular real money games include:

1. RummyCircle: It is a multiplayer card game played by 2 to 6 players. With over 5 billion games played in 2023 alone, RummyCircle brings players the thrill of competition and the joy of winning.
2. My11Circle: It is a fantasy sports app that allows users to play fantasy cricket, football, and kabaddi games and win cash prizes. It has made use of AI models to offer a personalized experience since its launch in 2019. In 2023 alone, over 500 million fantasy teams were created, rewarding players for their knowledge and passion for their favorite sports.

‍

The company is now expanding to more board and multiplayer games. They are actively investing in cutting-edge technology and startups to further its aim of creating more immersive and social experiences for its gamers.

The Data Science team wanted to be self-reliant

When we started working with the team at Games 24x7, The team had already been serving models to their millions of customers. However, to serve models at this scale they had to ensure the reliability and performance of models before each release. Since the team did not have self-expertise in handling infrastructure and doing production-ready testing and deployment they had to depend on the engineering team for the following:

1. Load testing: The team could only do minimal load testing themselves that involved running requests in a loop. The engineering team had to do the load testing themselves and provide feedback to the Data Science team which then worked on the feedback and sent it to the engineering team again for testing. This caused weeks of delay in shipping the model.
2. Autoscaling: The team aimed to cut costs by implementing dynamic autoscaling based on traffic patterns, as the current practice of provisioning a large instance for expected model traffic resulted in unnecessary expenses during periods of lower traffic.
3. Provisioning Infrastructure and Deployment: The Data Science team followed a series of steps to deploy to model often directly from their local devices. The deployment also happened in a non-standard way across teams and often lacked good practices like version control, data lineage tracking, etc. 
4. Async Inference: The team repeatedly wrote sidecars for consuming requests from a Kafka queue. This was very time-consuming and required repeated effort for each project.
5. Implementation of a feature store: The data science team sought to implement a feature store for project reusability and tracking features across models. Although the data engineering team had a similar solution, its usability in machine learning projects was hindered by difficulties in direct interaction.

The platform and engineering team wanted more visibility

While the Data Science team wanted more empowerment and speed, the engineering team within the company wanted more visibility and control to keep the infrastructure cost-optimal and secure. Some of their issues they had been facing were:

1. Lack of standardization in deployments: Non-standard deployment within the DS team made them serve ad hoc requests for each project to be deployed. Also, when deployed, the models lacked good S.R.E. practices like versioning, git integration, checkpointing, data lineage information, etc.
2. Limited visibility into cost and performance of models: Since deployments were scattered across different infrastructure types (EC2, Sagemaker, Databricks) and there was no centralized repository of deployed resources, the engineering team had a hard time keeping track of the utilization of these resources, their performances and optimizing the costs.

Games 24x7 team became design partners with TrueFoundry

Looking at the requirements of the team, we proposed to build an ideal setup for the team that can solve the concerns of both the data science and the engineering team. However, the ideal system would have required additional developments of some crucial features by the TrueFoundry team as well. Some of these features involved the following:

1. Asynchronous inference service by simple config change: This would enable the data science team to directly deploy model services for their huge traffic loads without fear of dropping out on requests. We wanted to make this as simple as a simple toggle for the developers instead of having to write a sidecar themselves which had been taking a lot of their time and effort.
2. Autoscaling to 200 RPS without performance drops: The team aimed to implement autoscaling for efficient resource utilization without compromising model performance or introducing significant latency. Testing and analyzing autoscaling performance were crucial in making this decision.
3. Easy to use Load Testing: We aimed to offer developers a user-friendly load testing interface using Locust to simulate expected peak traffic. Initially, it would be provided as a simple script for data scientists, with a later option for a UI, reducing dependence on the engineering team.
4. Kafka Deployment and Integration: The team earlier did not have a dev instance of Kafka so they could not emulate production-like scenarios while doing post-development testing. We wanted to give the data scientists a simple method to deploy and start using applications like Kafka in development environments without having to rely on the engineering team.
5. Metrics tracking and alerting: The team wanted to track the performance of its model and trigger alerts whenever a resource or performance-related issue was anticipated. This would help both the DS and Engineering team take quick steps to fix any issues.

At every step of the journey of building these features, the Games 24x7 team was pivotal in testing what we built and providing us with critical feedback. This feedback has been pivotal in shaping the productization of these features and letting our other customers also be able to use them.

With support from the Games 24x7 team, we were able to build and ship these features to the team in less than a month! Throughout the process, the Games 24x7 team acted as partners to us.

We helped the team ship projects 3X faster!

When the development of most of the new features was completed, we helped the team put all of it together and deploy it on the TrueFoundry platform in a scalable fashion that was required for their traffic levels.

Some of the main value adds that the team derived from working with TrueFoundry were:

Data science team could save weeks by doing things independently

Using the new setup the Data science team could perform a lot of tasks independently that they earlier had to depend on the Engineering team. Some of the changes included:

1. Being able to do load testing themselves
2. Being able to deploy Kafka in the Dev environment for testing
3. Being able to configure auto-scaling themselves

The engineering team got more visibility and control

The engineering team was able to have a finer view of the ML Operations by making use of the TrueFoundry platform. The platform helped provide the team:

1. Singular dashboard of all deployed models
2. Visibility and insights into resource utilization by different projects
3. Cost reduction by optimizing the resource allocation

Platform ensured standardization and SDE best practices

By making use of the platform the team was able to create a standard deployment model with which any model within the organization can be deployed. It homogenized the deployment process for which earlier the engineering team had to spend time separately for each project hence causing delays.

Due to the design of the platform as soon as the team started deploying with TrueFoundry some of the SDE best practices were auto-enforced:

1. All code is versioned on bitbucket: The platform auto-dockerized code from the required commit and deployed it. This ensures that all the code is versioned.
2. All previous versions can be redeployed: The platform saves all previously deployed model versions so that any of them can be redeployed to reverse any changes.
3. Maintaining data lineage: All access to data and data features are logged onto the platform and can be retrieved and queried as needed.
4. Storing artifacts: The platform gave the team much more freedom to log any artifacts that they wanted with proper versioning.

The Games 24x7 team helped us build new features at lightning speed

Working with the Games 24x7 team helped us build some of our most used features today at a lightning pace that helped us deliver a production-ready version of these features within a month. This includes features like:

1. Asynchronous Inferencing
2. Automated Load testing for Models
3. Integration and deployment of Kafka
4. Metrics and Alerting

We continue to build working with the Games 24x7 team and with this continued partnership we are trying to build a product that is loved by both their Data Science and Engineering teams.