Dynamic AI Project Estimation

Author(s): Stavros Theocharis

Originally published on Towards AI.

Mastering the Basics for Agile Success

Estimations

Nowadays, estimations are used by almost everyone. The customers need them to both plan and control when they will start using the results of the project. Estimations also help the project manager figure out the scope, total amount of work, and rough cost of each task or the whole project.

Estimation is helpful in a number of situations, such as:

• Knowing how work is set up: Split a task into several smaller tasks to see the main steps you need to take.
• Understanding complexity: It’s hard to give an estimation for a complex task by itself, but it’s easier to give an estimate for each part of the work structure. It helps to figure out how hard the job is and how long it will take to finish.
• Figuring out costs: In most businesses, you won’t be able to start working on a project until you’ve explained and defended how much it will cost and what resources it will need.

What is the reality?

Let’s face a specific truth…. Estimations, in general, don’t work, and this is their biggest issue. Our plans are wrong, not complete, and often have nothing to do with how the work will actually be done. Even if the team has faced a similar task many times, it’s hard for even the most experienced software developers to guess how many hours it will take to finish it. You need relevant samples and a good estimation process to use relative estimates in AI projects. Relevant estimates can, for example, be some simple statistical estimations that take the average length of all relevant tasks done in the past. To make this kind of estimator, we must first gather a set of data.

The trick, as mentioned above, is to break down each task into its own level. Because of this, Scrum says you should use relative story points instead of exact hours. First, your estimations become more accurate at the level of the task, then at the level of the sprint, and finally at the level of the project. If you don’t have any past experience that can help you make relative estimations, you should only make an absolute estimation for the first sprint of the project. From here on, you should use the tasks you’ve already done to make new estimations.

Business and implementation estimations

Estimations can be identified from two different points of view. The first point of view is the implementation part, which is mostly understandable to both project managers and team leaders who are willing to get the project done. At this point, the main goal of the estimations is to better understand the time and money needed to build the final solution.

The second point of view has a lot to do with the business goals of the project. This point of view is usually not visible to the implementation team. Most of the projects usually have a business model to set goals for increased revenue, making customers happy, lowering costs, and so on. When someone is proceeding with the implementation estimations, this business model should be taken into account because it somehow sets the possible restrictions. In AI projects, business estimations can be made by first taking into consideration the business model’s budget constraints and then making a set of business metrics that can be used to calculate how well the project is performing.

You should start by laying out the general steps, and then go into more detail. If you’ve done projects like this before, you can look at their plans to figure out where to start. Collecting and using data from other projects can be a great way to get estimates, so don’t discount your past work.

When your outline has enough information, it’s time to make a proposal for the software architecture. This is a very important step because it’s not always possible or even cost-effective to match up outlines with customer or end-user needs. You should have at least a general idea of what technologies you will use, how they will work with the rest of the customer’s system, and how your solution should be deployed. If there are important non-functional requirements, like being available 24 hours a day, 7 days a week, the software architect should also think about how to make them happen in terms of technology and system design. Putting together a high-level architectural vision will help this outline make sense. If you think the outline needs to be changed, don’t be afraid to do so. Software design is a difficult task that should be done by an experienced engineer. If you don’t have a lot of experience designing software solutions, ask someone on your team for help, or even better, make software design a group effort.

Once the outline is done and you have a plan for the software architecture, you can start estimating the project. Simple statistical methods like the Program Evaluation and Review technique to estimate numbers (PERT) can do the job.

What is PERT?

PERT, known as the Program Evaluation and Review Technique, serves the purpose of estimating the anticipated duration required for task completion. This method proves exceptionally valuable when dealing with scenarios featuring an abundance of unfamiliar variables. Initially crafted within the United States Navy, PERT has since gained substantial traction among numerous project overseers.

Within the framework of PERT, there exists the capability to assign a rating to each task, utilizing a range from 1 to 3. Specifically, this involves contemplating the time investment a task would demand under optimal conditions. Furthermore, the potential for minor technical glitches or complications arising from prerequisites that hinder the progress must also be factored into the developmental trajectory.

Therefore, the primary factors that necessitate computation encompass the following:

1. Most likely estimation: This entails providing the most precise assessment feasible for the given task.
2. A pessimistic estimation: This represents the duration required for task completion in the event of adverse circumstances. This category encompasses higher risks, such as unsuccessful experiments and intricate debugging sessions.
3. An optimistic estimation: This refers to the anticipated time span to accomplish the task should conditions proceed optimally.

PERT estimation = (optimistic estimation + 4 x most likely estimation + pessimistic estimation) / 6

PERT standard deviation = (pessimistic estimation — optimistic estimation)/6

In order to have a confidence interval of about 99.7% (be sure of the extent of probability) you can calculate: Pert estimation ± 3 x PERT standard deviation (3 can be set to 2 for restricting the range).

Do not forget to use data from projects that are already done as a starting point for relative estimation. When estimating, the more outside sources you use, the more accurate and risk-averse your estimates will be.

Goals of estimations

While formulating projections, maintaining a steadfast focus on the ultimate objective holds the utmost significance. Crafting and sustaining accurate estimations demands a substantial investment of effort and time. Particularly within the realm of AI ventures, prognosticating outcomes proves intricate due to their intricate nature, thereby amplifying the significance of garnering trust from both your team and clients in these forecasts. This trust is pivotal as it correlates positively with the likelihood of successful execution.

It’s worth noting that the level of certainty tied to estimations diminishes when confronted with uncharted territories, such as crafting solutions for novel business domains or the integration of pioneering algorithms and technologies. In these instances, possessing a coherent roadmap toward the culmination becomes a guiding beacon.

However, it’s judicious to avoid excessive reliance on exact calculations pinpointing the temporal aspect or meticulously detailed blueprints. Instead, a discerning application of estimates is advisable. They serve as tools not only for ensuring congruence between implementation strategies and customer expectations but also for steering projects in alignment with these desires.

Risks

In the realm of AI projects, the initial and pivotal point of vulnerability lies in delineating the project’s objectives. The main point of success in a good understanding of the desired outcomes. It is very interesting when the project implementation begins in a quick way, but if there is a lack of precision, this can diverge significantly from the actual business requisites. Hence, it is very important to have well-established and well-defined objectives. These objectives can operate like a compass for the team, enabling them to discern between accurate and erroneous resolutions.

How to approach research projects?

A research endeavor encompasses any project geared towards unraveling solutions to previously unencountered predicaments. Notably, the motivation behind research projects extends beyond the enhancement of scientific knowledge. They encompass scenarios where teams delve into unexplored realms, such as novel business domains or emerging AI libraries, and strive to uncover innovative applications of AI within business contexts. This holds true for almost every AI initiative, wherein a subsidiary research project takes charge of the modeling process.

Nevertheless, a fundamental issue with research projects is their tendency to lack comprehensive breadth. Each research undertaking mandates a well-defined objective; without such clarity, successful culmination remains elusive. Moreover, addressing external challenges holds equal importance in the realm of research endeavors. The financial allocations for research naturally escalate in proportion to the expansiveness of the project scope. Budget constraints, if present, can also impose limitations on the depth and duration of the research.

Once the quantum of research feasible is determined, collaborative efforts can be directed towards tackling the pending experiments. Each entry on the backlog should embody an idea capable of enriching model quality or facilitating the realization of desired functionalities. The SMART criteria — specific, measurable, achievable, relevant, and time-bound — serve as a pertinent framework for articulating the specifics of each experiment.

Conduct swift yet comprehensive quality assessments on all ongoing experiments within your current research cycle. This practice aids in gauging the anticipated time investments for each experiment. To safeguard the trajectory of your research projects and prevent potential missteps, it’s imperative to adhere to the following guidelines:

• Set an exact goal
• Set the criteria for success
• Define the limits, such as the time and money limits
• Fill up the queue of experiments
• Prioritise based on what you want
• Keep track of all the tests and their results
• Make the code easy to copy
• Write down what you find

Some additional parts that need to be considered are:

• Identify what the nice-to-have is and what is the main part
• Reduce effort for not important parts in the beginning
• Try to be as lean as possible (based on the budget you have)

A considerable portion of projects relies on a top-down estimation approach, wherein the process originates with the overarching budget allocation for the project and subsequently deconstructs it into smaller components. This method is particularly employed when the project’s financial framework and pricing are already established. However, this approach encounters a notable predicament: initial estimates tend to be less reliable at the project’s outset. The challenge arises from the fact that determining the required budget is intricate before a comprehensive understanding of the project’s requirements and a detailed plan are in place.

This challenge can be further exacerbated, particularly when gauging the initial quote’s accuracy against the eventual project outcome is unfeasible. The efficacy of top-down estimation is optimized when there exists a profound comprehension of how each task within the project’s scope contributes to the initial fixed price. Nonetheless, this level of understanding demands substantial time and effort, which is often not feasible to ascertain manually.

Some additional helpful parts

• No one can work on projects all the time because everyone needs breaks, goes to meetings, gets stuck in traffic, and so on. So, you should assume that a resource isn’t available 20% of the time but is available 80% of the time. If a project resource is only available half the time, set the maximum availability units to 40%. When setting up the schedule for your project, don’t forget to account for each resource’s holidays, plant shutdowns, training, and vacations. No plan ever goes exactly as planned. You need some wiggle room because some tasks will be late.
• Include a buffer: No plan ever goes exactly as planned. You need to have some margin for error because some tasks will be late. A good idea is to add a buffer task at the end of certain phases (like those that use new technology that your team has little or no experience with) or to extend the project’s overview deadline for that phase by 20% from its original length. For instance, if the original length of a phase is 100 days, you could make it 120 days.
• Try to identify some of the possible risks at the beginning of the project and make this a continuous procedure.

Conclusions

• Estimations are used to plan and control projects, understand complexity, and figure out costs.
• Estimation is difficult and inaccurate, so a relevant sample and a good estimation process are needed.
• Use relative story points instead of exact hours to improve accuracy.
• Differentiate between business and implementation estimates, and consider the business model’s budget constraints.
• Outline steps, make a proposal for software architecture, and use statistical methods like PERT to estimate projects.
• PERT involves rating for each task on a scale of 1–3 (best case, most likely, and pessimistic estimates).

References

Dubovikov K. (2019). Managing Data Science. Packt Publishing

https://projectmanagementacademy.net/resources/blog/a-three-point-estimating-technique-pert

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI