Our client lost only a quarter of the budget they dedicated to an AI project because they chose to start with a proof of concept. The PoC allowed them to test their idea and fail fast with limited spending.
To avoid wasting time and effort, always ask your AI solutions consultant for a proof of concept – especially if your company is just testing the artificial intelligence waters.
This article explains what an AI proof of concept is and elaborates on the five steps that will guide you through your first PoC, together with the challenges that you might encounter on the way. It also presents AI PoC examples from our portfolio. And you will find a happy ending of the example depicted in the opening paragraph.
What is an AI proof of concept, and when is it essential for your project’s success?
An artificial intelligence proof of concept (PoC) is a prototype or a demonstration of a proposed AI solution designed to test whether the solution is feasible and likely to be successful.
The purpose of creating an AI PoC is to validate the concept, assess the proposed solution’s potential benefits, and identify any potential challenges or limitations.
An AI PoC typically involves building a small-scale version of the proposed AI solution and testing it in a controlled environment to see how it performs and whether it meets the desired objectives. The results of an AI PoC can then be used to inform further development and implementation of the solution.
Compared to ordinary software PoCs, an AI POCs may involve more complex considerations, such as the ability of the AI solution to learn and adapt over time, and the potential ethical implications of the solution, such as AI bias.
The technology stack for PoC AI projects is different, too:
- Machine learning algorithms. These algorithms allow the AI system to learn from structured data and make predictions or decisions based on that learning. There are many different types of machine learning algorithms, including supervised learning algorithms, unsupervised learning algorithms, and reinforcement learning algorithms.
- Neural networks. These computational models are inspired by the structure and function of the human brain. Neural networks can process and analyze large amounts of unstructured data. They can be trained to perform various tasks, such as image recognition, natural language processing, scenario modeling, and prediction.
- Robotics. This technology can be used to build physical systems capable of autonomous operation and decision-making. Robotics solutions incorporate sensors, actuators, and other hardware components that allow engineers to build a robot that can interact with its environment and perform tasks.
- Cloud computing. Cloud computing platforms like Microsoft Azure, Google Cloud, and AWS provide the computing power, storage resources, and pre-configured services needed to support the development and testing of AI POCs. These platforms can also host and deploy AI solutions once they have been developed and tested.
Creating an AI PoC involves gathering and preparing data, building and training machine learning models, and testing and evaluating the performance of the AI system.
The time it takes to create an artificial intelligence proof of concept can vary widely depending on several factors, including the complexity of the proposed AI solution, the resources and expertise available for the POC, and the specific objectives of the POC. Some AI POCs can be developed in just a few days or weeks, while others may take several months or even longer to complete.
When is there absolutely no way around the AI PoC?
It’s essential to start your project with a PoC in the following scenarios:
- Your project relies on an innovative idea that was not tested before. Something that was studied at the business level, but not attempted technically. Neither you nor your tech vendor are confident if this can be implemented at all.
- If you need to demonstrate to stakeholders, investors, etc. the feasibility of your idea within a limited timeframe. A PoC will do the job better than an interactive prototype or something similar.
Are there situations where an AI PoC is a waste of time?
Even though an AI proof of concept is beneficial in most cases, there are a few exceptions. If your project falls under the following categories, PoC can be an overkill:
- If your idea and approach are exceptionally well documented from the functional and technical perspectives. This is rather rare.
- If the solution you want to develop is standard and resembles common practices in the field. We already know that this is feasible and possible from the technical perspective.
- If you want to build software that your front-end and back-end developers understand and have already worked on something identical before
Which benefits can you gain from AI proof of concept?
Using AI proof of concept brings about the following benefits:
- Identifying potential challenges before committing more resources to this endeavor. PoC AI allows you to “fail fast, fail better.” If the team encounters challenges they can’t overcome, all stakeholders have time to regroup or maybe change the hypothesis to reach the same goals through other methods.
- Minimizing business risks, as you test innovative ideas in small steps instead of diving into a long-term project
- Improving data collection practices
- Getting investors and other stakeholders on board
- Saving time and resources. AI PoC might uncover business or process-related issues and give you time to fix everything before starting a full-scale project
- Building expertise and creating knowledge owners who will mentor other team member on similar projects in the future
- Testing the tech stack on a smaller scale to understand its suitability for the selected business case
Examples from our portfolio where AI PoC saved the day
Here are a few AI PoC examples from the ITRex portfolio that will help you appreciate the proof of concept approach even more.
Realizing that ML alone is not the answer
A large cargo logistics company performs 10,000-15,000 shipments per day, and every shipment is accompanied by bills of lading and invoices to cover the operations. Employees were exhausted by handling all the documentation manually. The company wanted to build an ML-powered solution that would use optical character recognition (OCR) to process scanned documents and identify different fields.
The client believed that machine learning was the best choice for this case, so we proceeded with an AI PoC to test this assumption. Soon we realized that the documents were formatted differently, and the labels used for fields varied significantly. For instance, the Load ID field alone had 8 aliases. As a result, the ML model kept growing. It became slow and inefficient. Our team decided to accompany this model with a dynamic algorithm (e.g., a dictionary where different field labels are hard-coded). This modification improved the solution’s performance significantly and saved the client time and money.
If the client had decided to skip the AI proof of concept, they would’ve wasted seven months just to realize that their initial idea of a pure ML-based model was not the optimal solution here.
With the artificial intelligence PoC, they came to this conclusion in merely two months. Since the successful completion of the AI PoC, we built an MVP that could handle four types of documents, taking over around 25% of the manual processing load.
Taken aback by Meta’s restrictions on data usage
A client in the entertainment industry wanted to build an AI-driven analytical platform for independent musical performers. The solution was supposed to crawl social media, including Facebook and Instagram, to gather data. It would process all this information to gauge people’s sentiment towards the artists. Musicians could sign with the platform and receive feedback on what social media behavior is the most beneficial for their success.
We proceeded with the AI proof of concept to test the idea. After just two weeks, we realized it was simply impossible to gather data from Facebook and Instagram to use it for the purpose described above. Typically, some of the data can be retrieved via Graph API. Combining this with a verified business account in Meta, we assumed we would gain access to the required information. However, the client couldn’t supply us with a verified business account, and the data from Graph API alone was not sufficient for this solution work.
If the client had decided to skip the PoC, they would’ve wasted around $20,000 on the discovery project.
This would include a detailed description of the solution and the estimation of the development costs. But as they chose to start with the AI PoC, they spent only around $5,000 before figuring out that this idea was impossible to execute due to data access restrictions enforced by the Meta company.
A five-step guide for your first AI proof of concept
Here are five steps that you can follow to successfully go through your AI PoC. In this section, we also list challenges associated with each step.
Step 1: Identify which issue(s) you want to tackle with AI
It is essential to specify what exactly you want to accomplish by implementing artificial intelligence PoC. The selected use case needs to be of high value and represent something that you can address best with this technology. If you have doubts, a good place to start is to look into what others in your field are using AI solutions for. Another way to go is to investigate the problems that your business is facing and compare it against the potential of AI.
After you’ve accumulated a list of opportunities, you can ask the following questions to determine which ones are the best fit for your project at the moment:
- Is the problem you intend to solve specific enough? Can you evaluate the results to determine success?
- Did you already attempt to solve this problem with other technologies?
- Do you have the talent and the funding to support this project until the end? If there is no suitable in-house talent, can you hire an external dedicated team?
- How will it impact your business? Is this effect significant enough to put in the efforts?
- Will you be able to sell this to the executives? Is your organization ready to take on such projects?
- Does your firm already have a data strategy? How will it align with this project?
- What are the potential risks and limitations of using AI to tackle this problem?
Associated challenge:
- Selecting a use case that doesn’t add much value or doesn’t use the full potential of AI. Artificial intelligence is an expensive technology, and choosing an insignificant case will mean you spend more than you will receive. Check our article on how much it costs to implement AI to gain a better understanding of the expenses.
Step 2: Prepare the data
Now, as you have your problem clearly defined, it’s time to aggregate and prepare the training data for the AI algorithms. You can do that by:
- Checking which data is available for use within your company
- Generating semi-synthetic data using specific ready-made applications or your own solution
- Purchasing datasets from reliable providers
- Using open source data
- Hiring people to scrap the data that will fit your purpose
You don’t have to limit yourself to one source. You can use a combination of several options listed above.
Turn to data scientists to run the initial data screening. They will perform the following tasks:
- Structure the data
- Clean it by eliminating noise
- Add any missing data points, in case of tabular data
- Perform feature engineering (i.e., adding and deleting data fields)
- Apply manipulations, such as combining or filtering data
Data scientists can advise you on how to gather additional data or how to narrow the AI proof of concept’s scope so that you can achieve the desired results with the existing datasets.
When the data is ready for usage, split it into three sets:
- Training set, which the model will use to learn
- Validation set to validate the model and iterate on training
- Testing set that will evaluate the algorithm’s performance
Associated challenges:
- The training data is not representative of the entire population. In this case, algorithms might perform well on common cases, but will deliver poor results on rare occurrences. For example, a healthcare ML model that analyzes X-rays might excel at detecting common disorders, such as effusion, but will struggle to spot rare diseases, like hernia.
- Class imbalance, when the number of cases representing one class is significantly larger than the other, with a ratio of 99.9% to 0.1%
- Incorrect labeling, like mixing classes, e.g., labeling a bike as a car
- High noise in the training dataset
- Hard to achieve pure class separability. This happens when some data in the training set can’t be correctly classified under a particular class.
Step 3: Architect and build or buy the solution
You are probably wondering whether you should build the model yourself or you can procure an existing solution.
Here’s when it makes sense to create an AI model from the ground up:
- Your solution is innovative and doesn’t conform to an existing standard
- Ready-made solutions are costly to customize
- The closest off-the-shelf model is an overkill, and it does much more than you actually need
Consider procuring a read-made model if:
- The costs of buying and customizing the model are less than building it from the ground up
If you decide to build the AI algorithm from scratch, it will give you more control over its accuracy. It will take longer to complete the task, but it will be tailored to your business problem and your internal processes. You will not need to make changes to your system to accommodate external software.
Regarding the infrastructure for algorithm training and implementation, you can rely on the cloud instead of using the local resources. There are four parameters that you can consider deciding what suits you best:
- Security. If your data is very sensitive when it comes to security, then you better keep everything on-premises.
- Workload. If the processing load is rather heavy, opt for the cloud
- Costs. Evaluate what will cost you more – acquiring the resources locally or paying for the cloud usage over time.
- Accessibility. If you will only use the solution locally, you can depend on your in-house servers. If it needs to be accessible from different geographical locations, then it’s worth considering the cloud.
Every approach has its benefits and drawbacks. If you are operating in the healthcare sector, we have those clearly explained in the cloud computing in healthcare post on our blog. Otherwise, feel free to reach out to our AI experts to choose the best technology stack for algorithm training.
Associated challenges:
- Lack of proper training. This will cause issues, such as poor model generalizability, which means that the model can’t make accurate predictions on data that it has not seen in training. Coming back to X-ray image analysis in the medical sector, an algorithm might successfully analyze high-quality images captured by state-of-the-art scans but still fare poorly when applied to scans generated by older machines.
- Integration with existing systems, some of which may be outdated or powered by proprietary technologies
- Failing to come up with the suitable model architecture, e.g., being unable to pick the right ML model for the problem at hand
- The selected architecture’s capability can’t match the model’s requirements
- The input data is volatile, which means the model has to be frequently retrained
- Using more resources than your model requires to perform its tasks. There is no need to invest in a powerful server to run a simple model.
Step 4: Assess the AI proof of concept’s potential to generate value
This step is about evaluating whether the AI PoC can live up to expectations. There are several ways to perform the assessment:
- Go back to your key performance indicators (KPIs) and test the solution against them. These factors may include accuracy, customer satisfaction, speed, flexibility, fairness, safety, etc.
- Collect data on how your system operated before the AI proof of concept deployment. This would include the time spent on a particular manual task, the number of errors, etc. Next, you should use the information to evaluate the impact of the PoC.
- Compare the solution’s performance to other products that are regarded as the benchmark for this type of problems or the industry. For instance, a benchmark for image classification-related issues would be a model that delivers accurate results on popular datasets, like ImageNet.
- Gather user feedback either through focus groups or online surveys to gauge the levels of satisfaction and determine what is missing
- Conduct cost-benefit analysis to understand the financial impact of this solution on the organization
Associated challenges:
- Making a mistake in your assessment. It can be a simple math mistake during calculations, or an error related to estimating the business potential.
Step 5: Iterate on the AI proof of concept for better results, or scale it up
If the results that you received in the previous step were not up to par, you might consider modifying the solution and iterating the whole process. You can make changes to the ML algorithm and measure the performance with each adjustment. You can also experiment with different hardware components or alternative cloud service models.
If you are content with the AI PoC’s performance, you can work on scaling it in different directions. Here are a few examples:
- Apply the PoC to other business cases. Look for other applications of this new solution within your business. For instance, if you are testing AI as one application of predictive maintenance, you can try to apply it to other related scenarios.
- Scale the infrastructure. Review the technology used to run this software. Can you dedicate more processing power or more data storage capacity? Such modifications will enable you to use more data, decrease latency, and maybe deliver results in real time. It will also minimize the possibility of bottlenecks in the future.
- Optimize the AI PoC solution. Even though you already got reasonable results in the previous step, it might be worth looking for ways to improve accuracy. You can keep training your algorithms using new data, more accurately labeled data, etc. Or you can even experiment with implementing tweaks and changes to achieve better results.
If you decide to adopt AI company wide following the proof of concept phase, you can find helpful tips in our guide on how to implement AI in your organization.
Associated challenges:
- The architecture was not carefully considered. The solution might work well with 10,000 users but crash when the audience reaches 100,000.
- The model contains bugs that will manifest themselves when you attempt to scale the AI solution
- Applying the model to other business cases, other than the ones it was intended for. For instance, a solution that is meant to assemble a garden wheelbarrow can’t be applied to assembling trucks, as it might build a large garden wheelbarrow with a motor.
To conclude
When it comes to implementing AI, start small and stay manageable. Make sure you have a clear business case with defined objectives and metrics to measure success. And always consider creating an AI proof of concept, except for the cases presented at the beginning of this article. This will help you identify any potential obstacles before you go fully in and make a large financial investment in a solution that will not live up to expectations.
Do you want to implement AI in your organization, but aren’t sure if your business idea is feasible? Get in touch! Our team will help you conduct a PoC to test your idea on practicalities.
The post How AI Proof of Concept Helps You Succeed in Your AI Endeavor appeared first on Datafloq.
