Semantic Layer

Organizations today have the technical capabilities to collect vast amounts of business data. Data mining provides new insights into business processes and supports effective management decisions. However, large volumes of complex data require appropriate mechanisms for collecting, storing, and processing. Implementing these mechanisms may be hard without specific IT skills.

As a result, analysts focus more on obtaining, storing, and transforming data to make it suitable for analysis than on the analytical process itself. The semantic layer addresses this efficiency problem.

A semantic layer maps complex data structures to familiar business terms, such as product, customer, or revenue, to create a unified, consolidated view of information across the company. It lets business users access, manipulate, and process data through common business terms instead of specialized low-level languages. This allows specialists to focus on data analysis rather than data management procedures.

The word 'semantics' comes from Ancient Greek and refers to the meaning-creating relationships between words, phrases, signs, and symbols. For corporate data, semantics uses the relationships among schemas, tables, and columns in a data warehouse or data lake to create a straightforward business view. This view hides the complexity of the source information and presents dimensions, metrics, and hierarchies for analysis.

An intelligent semantic layer is an abstraction layer that facilitates a consistent way to interpret data. It maps complex data to familiar business terms, giving users across the enterprise access to the same trusted source of information and confidence in its integrity. Organizations can collect all definitions and business logic in one place and then manage and modify them in a centralized manner. The primary purpose of a semantic layer is to make data more useful to the business and simplify queries for users.

The semantic layer stores business terms as objects that users access through business views that the entire organization shares for data analysis. This data comes from various sources, including consolidated ones, such as warehouses and data marts, and local ones, such as OLTP and accounting systems, documents, and others. In simple terms, a semantic layer is a business intelligence technology that frees users from routine data management tasks and helps them solve decision-support problems more effectively.

As shown in this schematic, the semantic layer sits between the data warehouse level and user applications. It provides a simplified and consistent view of information regardless of the complexity of the underlying sources. It serves as a logical layer:

• Maps physical data structures to elements of their conceptual model.
• Defines all rules and relationships among data elements.
• Provides a common business glossary for the data.

As a result, business users can easily work with data even if they lack specialized technical knowledge of its sources.

Semantic layer levels

A semantic layer typically consists of several levels, each with its own functionality. The set of levels can vary depending on the task, data sources, and applications. A complete, universal semantic layer that supports all possible use cases should contain the following four levels:

• Data modeling layer. Defines metrics and data models so that all applications receive consistent and coherent information.
• Access control layer. Controls access to ensure that each end user can access only the appropriate information.
• Caching layer. Buffers data sources to support high concurrency and prevent delays when applications retrieve information.
• Interface layer. Provides compatibility between different data sources and downstream applications.

As mentioned above, this set of layers may vary. For example, local data sources may not experience the significant transmission delays associated with wide area networks, so they may not require a caching layer. Similarly, if applications control access, the semantic layer does not need to provide access control.

Main implementation approaches

The concept of a semantic layer is almost as old as BI tools themselves. In the past, each tool had its own semantic layer, used by specific teams within an enterprise. At that time, data sources were mostly disparate, and data volumes were moderate. As data volumes grew, enterprises began consolidating data on modern platforms, while business users kept using familiar business intelligence tools.

Maintaining multiple versions of business logic in each BI tool often caused users to interpret the same data differently. Organizations therefore needed to create a unified view for both analytical tools and business groups.

Companies use many different approaches to implement a semantic layer. They may organize it for BI applications, data warehouses, or data pipelines, or create a universal layer that adapts to any task and application.

Semantic layer for BI tools

Most BI tools allow analysts to define their own semantic models: dimensions, measures (facts), and hierarchies. One option is letting business users create their own semantic models with the tools they already use. However, this approach makes it difficult to establish a single source of truth. Different teams need a shared data view that lets them access their information through common business terms.

After an organization creates a semantic layer for a BI application, all business users of that application can access any model.

Universal semantic layer

Unlike the previous approach, which creates a separate semantic layer for each application, this approach provides a single layer for all BI applications used by the company. All business users can access the same model regardless of which business intelligence tool they use.

Semantic layer in a data warehouse

This layer consists of metadata and descriptions that provide context and meaning for the data in the warehouse. It provides efficient access to the warehouse contents and simplifies their analysis and interpretation.

A semantic layer for a data warehouse describes the data structure and sources, the history of data changes, access rules and restrictions, and the subject area. The semantic layer helps users quickly and accurately find the required information, identify relationships among business objects, and make informed decisions based on analysis.

Types of semantic layers

The implementation approach and the information structure a semantic layer supports determines its type:

1. Data store, or "fat" semantic layer. Supports data warehouse operations.
2. Thin, or virtual semantic layer. Contains no data and stores only rules that describe its logic. Works with virtual data warehouses.
3. Hybrid semantic layer. Supports both data warehouse and virtual modes. Developers can specify which tables in the semantic model the system stores and which tables remain virtual. This approach can help balance performance and complexity. In some cases, the data may be too large for the semantic platform to store, making the hybrid model the most attractive option.
4. Metasemantic layer. Represents a relatively new approach. Its key feature is an open architecture in which developers describe metric definitions in a platform-independent language, ensuring high portability of semantic models.
5. Universal semantic layer. Forms an independent layer between raw data and information consumers. In this case, semantic models provide predefined data views that abstract away complexity and use business-oriented definitions. This type can also define a dimension hierarchy.

These semantic layer types focus on business intelligence and data mining tasks in decision support technologies for business management. However, organizations can also use them in other areas of IT, so other types may also appear.

Advantages and disadvantages of using a semantic layer

Using a semantic layer provides the following advantages:

• A semantic layer allows organizations to use multiple tools, platforms, protocols, and other technologies for data analysis instead of relying on a single BI application.
• A single source of truth creates an information architecture in which the system processes and modifies each data element in only one place. This eliminates discrepancies and conflicting interpretations among business users.
• A semantic layer improves analytical query performance and reduces computational costs.
• A semantic layer provides security: because it sits between the data platform and analytical tools, it protects information through authentication and authorization.
• A semantic layer increases the level of self-service analytics by giving all employees access to data regardless of their technical expertise.

However, a semantic layer can also present several challenges.

• Organizations incur additional maintenance and synchronization costs when the data infrastructure changes.
• Failing to understand the goals and objectives when designing and building a semantic layer can reduce data quality and impair analysis results, leading to poor management decisions.

Companies currently invest heavily in creating and maintaining data warehouses and purchasing various tools for business analysts, data analysts, and application developers. A semantic layer helps justify these investments by giving more users access to more information and increasing the number of people who make data-driven decisions.

Further reading:

• The DataOps Approach
• AI + BI: What Actually Changes