Theory of constraints: how to expand bottlenecks in data analysis

In the 1980s, Israeli engineer, physicist, economist and writer Eliyahu Goldratt created the Theory of Constraints, which bears his name. Let's analyze what the meaning of the theory is and why it is relevant for modern data analysis.

The Essence of the Theory of Constraints

Intuitively, it seems that in order to optimize the operation of a system or process, it is necessary to comprehensively influence all of its elements or its stages — ie: try to improve everything. But, Eliyahu Goldratt proposed a fundamentally different approach. His theory of constraints is a system management methodology based on the search for single, yet critically important parts of the process (bottlenecks). These parts determine the success and effectiveness of the entire system as a whole.

According to Goldratt's theory, the effect of managing a bottleneck far exceeds the result of the simultaneous impact on all or most of the problem areas of the system either at one given time or alternatingly. Optimization of a critical site entails a significant improvement in the business process as a whole.
Data Analysis Process.

The theory of constraints is universal. It can be applied to any system where there is a certain sequence of actions, a pipeline or a step-by-step process. Let's look at how it can be applied to business analytics. To begin, we will indicate what typical stages the business process of data analytics consists of:

1. Generation of an idea
2. Formulation of a business task
3. Implementation of the idea
4. Testing a hypothesis, model or process
5. Operational use

To determine the critical part of this process, it is necessary to identify a bottleneck. In the theory of constraints, there is a bottleneck marker — a stage in the workflow before which the most unprocessed tasks accumulate. Based on this feature, it is possible to determine what is the bottleneck in data analysis.

The stages of generating ideas and formulating business tasks usually lack fundamental problems. But, as soon as it comes to implementing an idea that needs to be performed by programmers, difficulties tend to arise.

The shortage of developers -combined with a continuous stream of new problems- leads to the fact that the list of overdue tasks in the IT department is constantly piling up. Programmers physically lack time to solve them — what we get here is the classic bottleneck marker.

This situation is typical for the IT departments of around 95% of companies. There are objective reasons for this.
About 45% of large companies around the world are experiencing a personnel shortage in the IT field. The problem is further aggravated by the fact that more than 90% of the tested hypotheses are unsuccessful. But, unfortunately, the understanding of this comes only after their implementation. According to statistics, only 7% of analytical projects pay off.

Finally, we must take into account that approximately 80% of the time allocated to a given project pertaining to business analytics, the analysis itself is carried out alongside the collection, organization and cleansing of source data.

It is followed by the already scarce programmers who are forced to test hypotheses- 90% of which will be discarded- and at the same time spend 80% of their time preparing data for analysis.
Various tools and approaches are used to expand the bottleneck and increase the productivity of developers.

They are:

• Ready-made libraries, templates and patterns
• Scrum and Agile project implementation methodologies
• DevOps, ModelOps and CI/CD for automating the assembly, testing, configuration, deployment and integration of software

Thanks to this, the efficiency of the programmers' work is noticeably increased — but the problem of total shortage is not solved.

Low Code to Eliminate Bottlenecks

Another option for expanding the bottleneck may be the low-code approach. It means the implementation of typical tasks not by programmers but by ordinary users. With the help of visual design, the latter can independently solve non-trivial business tasks. This method does not require deep knowledge of programming. Low code lowers the requirement standards and therefore enables you to hire or employ almost any employee to solve the problem.

The simpler the tool, the easier it is to find and/or train specialists. The most obvious thing is to get employees of your own company, who already use Excel as the main analytical tool- to practice it. The specialists working in the company know the business well. It is much easier for them to analyze data than for outsiders.

It is also a good idea to get analysts within the team involved because much less work will be sent to the trash bin. Experienced business users can sensibly assess the adequacy of ideas due to knowledge of the subject area. They are able to find insights and clues in the data as well as quickly generate new hypotheses.

In any case, by hiring staff or training your own team, you can remove most of the tasks from developers.

Thanks to this, the bottleneck will be expanding — which, according to the theory of constraints, will significantly increase the efficiency of the entire system.

The additional value of low code is that it increases the involvement of business users who will be able to independently test various hypotheses. Routine operations in analytics (data collection, organization and cleansing) will be solved by civilian data scientists, without distracting programmers. As a result, developers will focus on solving genuinely difficult tasks.

Low code is applicable for solving a variety of advanced analytics tasks. A typical low-code tool is the Megaladata platform. It is a software product for implementing the entire analytical process: from integration and data preparation to calculations, modeling and visualization. To sum things up, we can say that low code is not a replacement for development but the best outright investment in digitalization because:

• The bottleneck of the data analysis process is expanding by attracting business users
• The involvement of specialists who are not familiar with programming increases: they can solve routine tasks, test hypotheses and identify patterns
• Scarce programmers are attracted to complex tasks that cannot be solved without them
• Complex blocks are implemented by developers in code and embedded in data processing scenarios

The low-code approach is not just a convenient tool for a wide range of users. It is also the most affordable option to increase the efficiency of the entire data analysis process.

With all that said, be sure to contact us if your company is interested in implementing a low-code tool for advanced data analysis which uses the Megaladata platform.