A Systematic Approach to Choosing the Best Technology/Vendor: MLOps version

Author(s): Eyal Trabelsi

Originally published on Towards AI.

The Ultimate Shopping Spree in the ML Shop

Machine learning is becoming an integral part of every aspect of our lives. A while ago, I wrote about a Systematic Approach to Choosing Technology/Vendor.

But choosing an MLOps solution is a bit different due to the long life cycle and the lack of mature tooling. Are you struggling to choose the right MLOps solution for your business? This post is for you.

Simple Recipe

To simplify the process of choosing the right vendor/technology, I propose a straightforward recipe:

1. Start by mapping your needs, resources, and constraints.
2. Search for potential solutions and eliminate any that don’t meet your requirements.
3. Evaluate the remaining promising solutions.
4. Make a decision — to go or not to go?

To be honest, it's generic for all technology in general, but in each step, I will give the ML perspective.

In the next section, we will drill down to map your needs, resource, and constraints.

Step 1: Map your needs, resources, and constraints

The first step in choosing the right technology or vendor is to map out your needs, resources, and constraints.:

The requirements and the needs depend on the component/functionality you fill in the ml lifecycle and can be seen in the following table.

The resources you have can also impact what solution you pick:

• Total ownership cost and Cost of mistake.
• Time to market and Maintenance time.
• Team expertise and Team size. This includes How many DS, MLEs, and Devops you got.

In terms of constraints, there are some that are generic, and some depend on the component/functionality you fill in the ml lifecycle.
The generic constraints are Existing Integrations, Change of Ownership, Specific Cloud/On-prem, Multi-tenancy, Compliance & licenses, Security, and Availability.
The per-component constraints are:

In the next section, we will drill down to searching for potential solutions.

Step 2: Search for potential solutions

The second step in the process of selecting the right vendor/technology is to search for potential solutions. However, before embarking on this search, it’s essential to familiarize yourself with technical terms and buzzwords to ensure that you can effectively evaluate potential options.

Once you have this foundational knowledge, you can begin researching potential technologies and vendors and keep a few promising options.

Then curate a short list of available options, the best way to focus is to take into consideration the following:
curate a short list of the available options.

• Filter aggressively on your constraints, for example, if you are looking for an on-prem solution or you got an unbalanced dataset, it will filter most of your options.
• Whether to buy vs build your own solution:
  – Build if it's easy! both to build and to maintain.
  – But when it's ROI positive and the shelf is not empty.
  – Skip if it's not ROI positive.
• Whether to choose an all-in-one or best-of-breed approach:
  – All-in-one is awesome for common and easy cases. Remember it's okay. Most of us are not Google.
  – Best of breed fit better in a more constrained environment (remember the lists from before?
  – You can even combine these, in cases most of your components are trivial and only a few are very constrained. Just make sure the integration is feasible within your resources
• Speak with the communities like MDLI, MLOps.community, MLOps TLV, and many more.

In the next section, we will drill down to evaluating promising solutions.

Evaluate promising solutions

To evaluate the remaining promising solutions, there are a number of steps you should take.

• Start by speaking with the vendor or users: This will help you gain a deeper understanding of the solution’s capabilities and limitations. Focus on stability and problems people face as most of these solutions are not mature.
• Do some POCS: Start with “Hello World” tutorials this will help you get a sense of how the solution works in practice. Do more through POCs if needed.
• Make a table for comparison: This can include interactions like releases, support, community, pricing, constraints, requirements, and resources.

In the next section, we will drill down to the decision.

Decision! Go? no, Go?

When you're deliberating about whether to proceed or not. Bear in mind that the most fitting solution might not necessarily be the perfect one.

It needs to suit your requirements, constraints, and resource. Strive to identify the solution that best serves your organization's needs, making the choice less intimidating. To make a decision about whether or not to go keep your future goals in mind and prioritize flexibility in your decision-making process.

It’s important to maintain a professional demeanor and avoid getting caught up in politics or letting your ego drive your decisions. Additionally, it’s important to be aware of human bias and take steps to counteract it.

Last words

In this article, we’ve provided a recipe for selecting the right ML technology or vendor for your business. By mapping your needs, resources, and constraints and searching for potential solutions, you can make an informed decision that aligns with your goals. Remember that there is no perfect solution, so focus on the fittest solution that fits within your context.

I hope I was able to share my enthusiasm for this fascinating topic and that you find it useful. You’re more than welcome to drop me a line via email or LinkedIn.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI