Automation and Data Analysis with Megaladata at a Large Industrial Enterprise

Team structure and focus areas

The company has a team that implements projects related to procurement, planning, and metallurgy. The main participants are managers and business analysts, with three developers providing support. The team has been actively using the Megaladata platform for a few years:

• The internal AI team solves 95% of tasks in Megaladata; 11 out of 18 people use the platform regularly.
• 2 large projects and several medium and small ones (about 10 in total) run on Megaladata.
• Some computer vision (CV) projects are embedded into the platform and into the control loop.

The use of the platform covers the following tasks:

• Hypotheses testing
• Operational decision-making
• Self-service analytics

This article looks at two of the company’s projects in detail: inventory optimization and procurement pricing.

Inventory optimization project

Inventory optimization was an acute task in the company's procurement system.

Project goal: Reduce warehouse inventory levels by blocking procurement of materials whose stock coverage exceeds 6 months.

The problem: Even minor issues required a large amount of resources, because analysts could not produce reports on their own.

When the company implements a new project, all these steps are necessary and inevitable, because they require a full development cycle. However, when the change is minimal, such as changing filter conditions or adding a calculated column, this process becomes too costly. The goal was to delegate these simple tasks so that business analysts could solve them independently instead of waiting for help from the IT department.

The company had specific requirements for a system that would let a business analyst resolve day-to-day issues without involving IT:

• Simple connection to specially prepared sources
• Visual tools for data preparation and algorithm development
• Big data processing
• A complete set of tools for automating daily tasks
• Teamwork capabilities

The idea was to implement a self-service tool that would shift routine, simple tasks from IT to the business and let analysts create sufficiently complex reports without preparing a technical specification and with minimal requests to IT. 

With Megaladata, the architecture of an analyst's request now looks as follows:

This project, which is now in the implementation stage, solves the inventory optimization problem as well as reduces the load on the IT department by assigning daily business tasks directly to analysts.

PRO-Market project

The company spends approximately $2 billion a year on procurement. Before any item is purchased, it has to pass a price evaluation. Run manually, this process used to take up to 40 days per cycle, repeating every six months.

The price evaluation process sets a base price that matches the market price. Procurement can only start after the Commerce department has completed the price evaluation. 

Project goal: Reduce the procurement budget by automatically extracting the lowest market price from external sources (price lists, trading platforms, websites, and data from other industrial corporations).

Smart search

The enterprise implemented the project, including all these solutions, on the Megaladata platform. In the process, ETL takes 80% of the time.

Let's look at the details of how the intelligent search subsystem (the core) prepares data.

The first stage is data consolidation. It consists of:

• Collecting the required data from different sources (export/integration)
• Creating a centralized database

The second stage is data cleansing. At this stage, the team identified and eliminated errors in items in the unified price information database to ensure further high-quality analysis.

The third stage is data standardization: bringing records from the unified database in line with company rules. This stage included:

• Converting data to a single format (normalization)
• Parsing strings into attribute/value pairs and weighting them
• Creation and enrichment of machine dictionaries (with synonyms, abbreviations, etc.).

The fourth stage is data search:

• Step 1. Mapping based on fuzzy search against a "flexible" set of record comparison criteria.
• Step 2. Calculation and assessment of record match according to the corresponding methodology.

The search stage consists of the following actions:

• Fuzzy search based on a configurable set of criteria.
• Setting up the criticality of presence and match, as well as the weight for each query attribute depending on the scenario in use.
• Mapping: identifying "candidates" from the unified database using value comparison algorithms and machine learning techniques.
• Match assessment: calculating similarity degrees and ranking the found items based on their match level.

The fifth stage of the intelligent search subsystem is decision-making: outputting the results (conditional price, market price, analogs, duplicates, best supplier, procurement anomalies) and loading confirmed results into the corresponding database.

PRO-Market project results

To assess project quality, the team prepared a random sample of 100 items for each of 10 inventory item classes, taken from different sources of price information. Experts received this sample for manual attribute extraction. In parallel, the team also submitted it to the system for automatic attribute parsing using grammars.

The comparison confirmed that the system's attribute parsing quality exceeds that of human experts by 12% on average. Moreover, the algorithm has no limits on the volume of data it processes.

Functional results of the project:

• Expansion of sources of price information
• Shorter time to calculate market prices through automation of manual processes
• Creation of a unified price information database
• Implementation of procurement process monitoring

Further reading:

• EMAIS Case in Poland: Automated Solution to Determine Current Market Prices
• AI Case Studies with Megaladata