The success of a project-based business largely depends on the visibility the company has over its projects, the current level of their performance and, of course, the factors influencing their health and success. That is why the ability to have an extensive picture of the way these factors interact with one another is totally essential.
Project managers put lots of effort into evaluating project success and making sure that each of their projects is on track. They need to analyze, predict, and prevent any possible factors that could adversely affect the project’s health in the future. Also, PMs need to be able to instantly provide relevant, up-to-date project data to leadership to support their decision making. So, clearly demonstrating the state of the project through structured visualized data is a crucial task. Moreover, this data can serve as a foundation for the lessons learned that could be further applied to some future projects.
We at ELEKS decided to create a system powered by advanced data analytics to offer our company’s managers a handy tool to support them in their daily work. Our aim was to design a custom metric system that could assist in supervising every internal project on a daily basis and carrying out early risk analysis.
In this article, you’ll find a story of how we created a solution with a BI dashboard called Project Health, aggregating and visualizing project data in a simple and transparent way. This tool not only enables managers to monitor and track projects easier but also allows the company’s senior management to view an extensive picture of project performance without having to deal with lots of project reports. The system’s dashboard allows one to control the project’s health level, get notifications — if it falls below the threshold, or drill down the details of the particular project.
Below, you can see a screenshot from the system’s dashboard with a sample project data:
This system is quite easy to use and understand as it is based on a simple 0 to 100 scale. Depending on the rating, the overall project’s success falls into three groups, each marked with a different colour (red, yellow or green). Later on in the article, we’ll dive into more details to show you how we built the in-house BI system.
How to get an extensive picture of the project’s overall performance
When it comes to measuring a project’s success, all criteria may be grouped into five well-known categories:
- Scope – to characterize the specific project goals, functions, tasks, deadlines, requirements
- Budget – to track the project’s costs usage
- Time – to define if the project is on schedule
- Quality – to assess the overall status of the project’s quality: process, testing procedures, issue identification, and resolution
- Resources – to evaluate project resource management.
It may be common practice to measure project success considering time and budget as the key parameters. But is it really the right method? Doubtfully. The fact that a project is finished on time and budget doesn’t mean it is a success. Obviously, such an approach doesn’t provide the full answer and does not address other factors contributing to the project’s success or failure. So, to provide a more extensive and up-to-date overall picture of the projects, we decided to build a system that could combine a comprehensive list of criteria in all five areas and present the results on the system’s dashboard.
So, how to measure the key 5 success indicators?
After deciding to use a combination of criteria, it was time to start building a complex math model capable of calculating and analyzing individual factors that affect the overall project health. To design the unique metric system, powered by Data Science, we implemented the Bayesian Network (BN), a directed probabilistic graphical model that deals with uncertainty and complexity. One of the most distinct advantages of this network tool is its ability to perform various types of analysis as well as to model complex relations between various faсtors. It can also perform some sophisticated interrogations of the system and include data from various information sources.
In the picture below, you can see a ready-to-use model based on data extracted and incorporated in the BN from the following sources:
- An issue tracking system used at ELEKS
- A team collaboration software at ELEKS
- MS Dynamics CRM
- Our proprietary HRM (a system for tracking human resource management processes)
- ELEKS’ code repository
- The internal financial management system.
A BN is structured into a directed acyclic graph, which consists of the set of variables (denoted by nodes) representing the probabilistic dependencies among those variables. The conditional dependencies in the graph are estimated by using the known statistical and computational methods. Constructing such a network requires expert knowledge of the underlying domain. So, at first, we built a directed acyclic graph and then included the conditional probability distributions for each node in the graph, taking into account the stakeholders’ experience and expectations.
How the model actually works
To explain how the model works, let’s model the following case. First, observations about the project are received, and then the state of the particular node changes and the reasoning process triggers the calculation of the probability values for all the dependable nodes. Here are some factors that cause reasoning:
- Customer satisfaction index
- Delivering a release on a specific date
- Changes in a cycle time value
- Changes in the number of issues that are in the ‘in-progress’ status
- Changes in the number of reopened bugs
- Changes in the ratio of closed and opened bugs
- Financial data in comparison to the company’s KPIs
- Project stage
- Project type
- Environment and hardware availability, etc.
The graphical model below demonstrates that the project’s success equals 87.36%, taking into account only expert knowledge.
But, as soon as the information is updated, it causes the value of the project’s success to change. Below, you can find various factors that might affect the overall project’s performance:
- The cooperation model used for this project is the fixed bid model.
- The project has reached a stable stage.
- The cycle time value is decreasing in comparison with the previous check.
- The number of the opened and reopened bugs is lower in comparison with the previous check.
- The project’s financial values fit the company’s KPI values.
- All the environment tools are set and in-use.
- There are no blocking requests concerning software and hardware.
- There are no open vacancies.
- The latest customer satisfaction rate is high.
Everything looks quite good, doesn’t it? But, here are some more factors, worth paying attention to:
- There is an increasing number of tasks in the ‘in-progress’ status.
- The predicted release date is currently out of the schedule.
Being out of schedule and having low performance is quite risky for project delivery, as it might radically change the level of the project’s success (especially when we are dealing with the fixed bid cooperation model).
All these new observations lead to the recalculation of the dependable states of nodes within the model, resulting in a new project success value – 12.71%. Fig. 4 shows how the model values have changed comparing to the Fig. 3.
How we proceeded to adapt our model for measuring project success internally
Well, our next step was to validate and implement the model within all the company’s projects. To do this, we questioned the company’s PMs about the status of their projects. Then, we compared their assessments with information that was obtained from the BN model and discovered that the results were pretty close to the project managers’ expectations. This process of validating the model unveiled the area for improvement. It concerned CPD (Conditional Probability Distribution) correction. Changing the CPD values helped us adjust the influence of one node to another, and as a result, we made them more or less sensitive regarding the changing observations.
And only after having conducted a thorough analysis, we proceeded in collecting data for all the company’s projects and then calculated their success. Now we have the system that is currently available for ELEKS’ senior management and project managers, offering them a new, better and advanced way to keep a finger on the pulse of each of their projects.
Figure 5 is the screenshot from the system’s BI dashboard, showing the state of the sample projects. Our aim was to simulate the real-life case:
What more are we going to do with our system?
The reports of a project’s performance commonly require from managers a great amount of their time and routine tasks. Fortunately, the system we developed allows one to track and measure the project’s success without any additional time required. Moreover, our solution assists in monitoring project-related risks and helps make well-informed decisions faster. So, the company’s PMs can significantly benefit from using the data-driven model, as it offers them extra visibility over the factors that are affecting their projects. The system also allows finding the cause of the issues faster, in the event that there are any. The company’s senior management is taking advantage of this tool as well, as it provides them with a more clear and extensive picture for each of their active projects. In other words, our solution saves the company’s top managers and PMs a lot of time and effort and definitely makes their lives easier.
And, of course, we are going to continue to improve the process of measuring project performance. Also, we plan to introduce our model to our customers, so they could use it as additional assistance to facilitate risk management, reduce costs and make better planning.
As we go further with the model enhancement and implementation, we will definitely write about our progress on the blog. In case you are interested in the topic or have any comments or thoughts to share, please drop us a line. We greatly appreciate any feedback!