Megaladata 7.3—A Major Release

New capabilities of working with data trees

The Tree to JSON component converts a hierarchical tree structure into a dataset where each row contains a JSON object. If the root node of the tree is an array, the component generates a separate output row for each element in that array. The node configuration wizard allows users to define root node processing logic, date and time conversion formats, and JSON naming conventions.

The JSON to Tree component converts JSON data into a hierarchical tree structure. It provides flexible parsing, with the option to create the tree automatically or manually. For easier management of large structures, you can load a tree pattern from an XSD schema or directly from a JSON file.

The new Tree Union component combines data from multiple input trees into a single output structure. It operates in one of two modes:

• Concatenate all trees: Consolidates root nodes from all active input ports into a single root array.
• Select the first active tree: Outputs data from the first input port that contains an active tree.

Note on Data Types: In Select the first active tree mode, the component checks all input ports for array definitions. If any connected input port has an array structure (even if currently inactive), the component forces the output into an array format. This ensures consistency when the node follows a Condition node where only one branch contains an active data tree.

The Tree Join component allows you to attach data from a secondary tree to a specific node within a main tree, creating an expanded hierarchical structure.

When configuring this component, you can attach the root of the secondary tree in one of two ways:

• As a child node of a container in the main tree.

• As a new element within an array in the main tree

New component: SQL Script

The new SQL Script component is designed to perform custom SQL requests in various DBMSs. It supports the execution of non-cursor-returning scripts and handles transactions. The component requires configuring a connection to a relevant database, similar to the Database import and export components. The component receives both tabular data and variables as input, and the output contains execution statuses and errors.

The node configuration wizard features an SQL editor with syntax highlighting and code completion. Input fields and variables can be used directly within your scripts as parameters or macro substitutions.

• Advanced configuration: A separate wizard page allows you to configure additional settings, such as execution timeouts or error suppression.

• Usage: This component integrates seamlessly into standard workflows. Common use cases include creating a new database table and immediately populating it, or executing a specific script for each row of input data.

The SQL Script wizard

Row Filter redesigned

The Row Filter configuration wizard has been completely redesigned. The main setup window is now a table, where conditions are set in rows. To set a condition, select a field in the first column, an operator in the second, and a comparison value in the third. Complex conditions are created using multiple rows, connected with the AND operator by default. If you need to use the OR operator, set it explicitly between the rows using the context menu or the toolbar.

A similar wizard was introduced in the Table visualizer's filter and the Data Filter page of the Database import node.

New Row Filter wizard

Changes to database-operating components

Connecting to a database without using a connection pool

When you configure a connection to the database, you can now opt out of using the connection pool by selecting Single session work mode. In this mode, the connection allows exclusive access to only one node at a time. When another node requests the session simultaneously, the error message Connection to DB is busy will be displayed.

When configuring the connection, along with the single session work mode, you can set a Session wait timeout (s).

With this setting, if a second node attempts to access the connection while it is busy, it will wait for the specified time. If the session does not finish within the timeout, the process ends with the 'Connection to DB is busy' error. The timeout ranges from 0 to 2073600 (24 days). You can also cancel the waiting process from the Processes pane.

Single session work mode specifics:

• Unsupported connections: Single session mode is unavailable for connections to MS Access, MS Excel, and other ODBC/Microsoft Office Access Engine connections.

• Connection behavior: The connection appears in the Processes pane only once. If the connection is lost, the system does not reconnect automatically.
• SQLite behavior:
  • Import: You cannot run multiple Database import nodes simultaneously. If a wait timeout is set, these nodes are queued and executed sequentially as the session becomes available.
  • Export: While the system allows you to configure parallel export branches without error messages, the Database export nodes will technically be queued and executed sequentially based on the specified timeout.

New features for working with ClickHouse

Support for ClickHouse clusters has been implemented. In the ClickHouse connection config wizard, a new Cluster parameter is now available. You can select the target cluster from the drop-down list.

When a Cluster is specified, the Database export component modifies its execution logic as follows:

• The CREATE TABLE request creates two tables:
  • A data table (suffix _data) using ENGINE = ReplicatedMergeTree for automatic replication in the cluster. If you execute INSERT INTO on this table, data is written only to the connected server.
  • A distributed table. If you execute INSERT INTO on this table, data is evenly distributed across the cluster tables.
• The DROP TABLE request is prepared using the ON CLUSTER {cluster} clause. If a distributed table is specified, the data is also deleted from the underlying source tables.
• The DELETE FROM request is converted to a TRUNCATE TABLE request.
• The ALTER TABLE ... DELETE FROM mutation (used for deletion by keys) and the TRUNCATE TABLE request are both executed using ON CLUSTER {cluster}. If a distributed table is targeted, the data is deleted from the underlying local tables. However, if the distributed table belongs to a cluster other than the one specified in the Cluster parameter, the execution will end with an error.

Features of importing from a database when working in cluster mode:

• If you import a distributed table {db}.{table}, the data is concatenated from all cluster tables specified in the distributed engine arguments: Distributed(cluster, database, table[, sharding_key[, policy_name]])
• If you import from the data tables {db}.{table}_data, the data is imported only from the server to which you're connected.

New settings in the ClickHouse connection wizard:

• Network send timeout (s): Sets the maximum time allowed for sending package data to the server. If the data cannot be written within this time, the connection is terminated.
  • Default value (in seconds): 120
  • Range: [0 to 2073600] (up to 24 days). 0 indicates an infinite timeout.
• Network receive timeout (s): Sets the maximum time allowed for reading incoming package data. If no data is received within this time, the connection is terminated.
  • Default value (in seconds): 120
  • Range: [0 to 2073600]

Changes in working with MS SQL Server

New settings added to the MS SQL connection allow you to establish a secure connection to MS SQL Server even if encryption is not explicitly configured on the server side. The default encryption settings are as follows:

• Encrypt the protocol: The default value is False.
• Do not check server certificate: The ceritificate is verified (the value is False). The setting is not displayed in the connection wizard if the Always verify all certificates option is enabled in the security settings (in the Administration section).

• The Encrypt the protocol and Do not check server certificate settings may be overridden by global OLEDB driver settings in the Windows registry or the specific driver version.
• Setting Encrypt the protocol to False does not guarantee a non-encrypted connection, as encryption might be enforced by the server or the Windows registry.
• MS SQL OLEDB 18 (and earlier): These drivers never check the server certificate, even if the protocol is encrypted.
• MS SQL OLEDB 19: This driver skips the certificate check only if both Trust Server Certificate is disabled in the registry and Encrypt the protocol is set to False in the application. In all other cases, the certificate is verified.
• Linux: When connecting without a native client, if Encrypt the protocol is enabled (even if the server doesn't require it), the certificate is checked. Note that self-signed certificates will fail this check.

When working in Megaladata on Linux, ODBC client connections are supported when the Do not use DBMS client option is disabled. The system uses MSODBCSQL (priority) or FreeTDS.

Important: When opening packages created in earlier Megaladata versions, the Do not use DBMS client option is reset to its default (i.e., disabled). This means MS SQL connections will stop working if the system lacks the appropriate ODBC driver (MSODBCSQL or FreeTDS). Additionally, when moving a package from Windows to Linux, the connection will automatically use the ODBC driver.

Available settings:

• If the environnment variable MEGALADATA_MSSQL_ODBC_ONLY is set to true or 1, the connection to MS SQL is performed through ODBC only.
• Using the environment variable MEGALADATA_MSSQL_ODBC_DRIVER, you can specify which ODBC driver to use to connect to the SQL Server. Possible values are libmsodbcsql and libtdsodbc.
  • If MEGALADATA_MSSQL_ODBC_DRIVER is not set, the driver is selected automatically (MSODBCSQL is the priority).
  • If the variable is set but the value is not recognized, a warning is logged and the driver is selected automatically.

Known restrictions when working through ODBC:

• The Lock timeout is not supported automatically. You can set it manually in the Database import script: SET LOCK_TIMEOUT timeout_period.
• The sql_variant type is not supported.
• For MSODBCSQL:
  • You can't connect to old servers without changing the security settings of OpenSSL (enabling protocols and algorithms recognized as unsafe).
  • If unixODBC is compiled without DEFAULT_ICONV_ENCODING UTF-16, requests using Unicode characters with codes above 65535 in will not be executed.
• For FreeTDS:
  • The Do not check server certificate connection option is not supported.
  • If Encrypt the protocol is disabled, the packet containing the login and password is not encrypted.
  • If the connection is lost it is not automatically re-established.

For MS SQL, support for the xml type when connecting via ODBC is added.

Within the same connection, you can now import strings with different codepages. The Codepage parameter in the connection config wizard is renamed to Message language.

Connecting extensions for SQLite

SQLite extensions are additional software components that add new features to the standard SQLite kernel. Many useful extensions can be found on specialized resources (e.g., https://github.com/nalgeon/sqlean or https://sqlpkg.org/).

To use these extensions, you must download and save them into the sqlite folder located next to the Megaladata executable file:

• Windows default path: C:\Program Files\Megaladata\Server
• Linux default path: /var/opt/megaladata/server

Other changes in working with databases

In the Database export component, you can now completely re-create tables. In the node configuration wizard, the modal window for table creation now has the setting Delete existing table. If it is enabled, running the node includes two steps:

• Removing the existing table (if it exists) using DROP TABLE IF EXISTS table_name (or the equivalent command for DBMSs that do not support IF EXISTS).
• Creating a new table through a relevant SQL query.

When the Delete existing table option is enabled, an additional read-only text block displaying the deletion command appears on the SQL query tab above the main table-creating query editor. For ODBC connections and MS Access, the delete query is performed in error-ignore mode.

Additionally, the Database export component now supports creating columns with the maximum possible size (mapping to types like nvarchar(max), ntext, or clob). The table creation dialogue offers a specific String(max) data type, as well as an All string(max) button, to convert all string fields at once.

When exporting to a database, you can now set key fields during table creation. The table creating dialogue in the Database export node wizard now includes options for selecting key fields and enabling a Not NULL parameter. Batch editing of fields is also supported.

In the Database import node's wizard, the Choose table mode now displays table field descriptions. (Note: This is not supported for SQLite and Excel connections.) Additionally, a Generate field captions from descriptions option has been added. When enabled, field captions, including those in the preview and filter settings, are populated from the database field descriptions. If a description is missing, the caption defaults to the field name. This setting is visible only if the database supports field descriptions.

Generating field captions from descriptions

In the Preview pane of the Database import node, it is now possible to copy data directly to the clipboard.

For MySQL, PostgreSQL, and ClickHouse connections, a Check server certificate setting has been added to the SSL settings. This setting offers the following options:

• Off: The certificate is not checked.
• Signature only: Only the Certificate Authority (CA) signature is checked (not available for ClickHouse).
• Full: Both the CA signature and the host are checked. This is the default option.

This allows you to configure verification using system certificates. To do this, leave the CA certificate parameter empty.

For packages prepared in earlier MGD versions, if no CA certificate was specified, verification remains disabled. If a CA certificate was set, the Signature only mode is used for MySQL and PostgreSQL, and Full mode is used for ClickHouse.

When exporting to PostgreSQL, the modes—Append data to table, Clear table and append data, and Execute TRUNCATE and refill with data—now provide a Use fast loader setting with the following values:

• Auto (default): The system automatically decides whether to use the fast loader. It will not use the fast loader for views (VIEW, MATERIALIZED VIEW), rule tables (RULE), or unsupported field types. If the system detects these conditions, it reverts to the standard INSERT method.
• Yes: Forces the use of the fast loader (using COPY BINARY ... FROM STDIN to write rows).
• No: Forces the export to use standard INSERT requests.

Note that for small datasets (less than 100 rows), the standard INSERT method is used regardless of this setting. When the Use fast loader option is enabled, we recommend increasing the Count of rows in a batch to improve export speed.

Added a prompt for login/password when connecting to the data store in interactive mode.

A configured connection can now be cloned.

Added support for Oracle Database 23. For OCI 23 and above clients, the BOOLEAN fields are now detected as Logical.

For MySQL, libmysqllicient 24 is now supported (when running Linux). Corresponds to MySQL Server 9.0.

Megaladata no longer supports Interbase connections. In existing packages, configured Interbase connections will be automatically replaced with Firebird.

Changes in working with web services

The Overall timeout (s) property has been added to the REST Request and SOAP Request wizards, replacing the previous data exchange timeout. Allowed values range from 0 to 2073600 (24 days). If the timeout is reached, an error is sent to the output port and processing terminates. By default, the timeout is unlimited (∞).

In the configuration wizards for REST Service and SOAP Service connections, the parameter Ignore SSL certificate errors has been renamed to Do not check server certificate. If the Megaladata security settings Always check all certificates or Always check HTTPS certificates are enabled, this parameter is hidden from the wizards and ignored during node execution.

When you configure the REST Service or SOAP Service connections, you can explicitly set the Basic authentication type. In this case, the first HTTP request sends the title Authorization: Basic base64(<username>:<password>). If the server's response is 401 Unauthorized, no attempts to use other authentication types are made.

Added support for NTLM and Negotiate authentication types for HTTP requests. Negotiate only works through NTLM, i.e. without Kerberos support. For the NTLM protocol, only the V2 version is supported .

When configuring a REST Service connection, you can now set the URL parameters type:

• Query string: The parameters are included in the query string as param-name=param-value (as before).
• Add segments: This method appends parameters directly to the query path. E.g., if the base URL is http://example.com/operation-path, running the REST Request node appends the parameter values in order: http://example.com/operation-path/value1/value2/....

• Replace: The parameters can be added to the URL line using special :param1 syntax. In this case, you cannot add parameters in the table.

REST Service connection parameters are defined exclusively during the initial wizard configuration. If the URL is changed dynamically via a variable, the parameters are not updated; the original names, types, and list of parameters remain in effect.

Empty parameter names are no longer supported for any type of URL parameter.

For the REST requests POST, PUT, and PATCH, you can also add URL parameters now.

In the settings of the REST Request node, new parameters have appeared:

• Wait between requests (ms)
• Repeat requests after an error
• Wait after failed request (ms)

These settings allow you to handle errors related to rate limiting (excessive requests per time unit) or server overload. Retries are triggered in the following situations:

• A network error
• The server returned 408 (Request timeout)
• The server returned 429 (Too many requests)
• The server returned any of the 5xx errors

If the request results in an error, an entry is added to the log. During subsequent retry attempts, failures are logged as warnings (warn) rather than errors. A full error is logged only if the final attempt fails.

New features of working with Python

You can now set up a Python environment (the name of basic container image) through the Python component parameters and node settings. New settings have been added to the component's parameters:

• Default environment (PY_ENV): Passes the PY_ENV variable to the python_run.sh script to launch a specific isolated environment.
  • Value: Selected from Known environments or entered manually (if Allow unknown environments is true).
  • Defaults: python (Linux); unspecified (Windows).
• Known environments: A predefined list of environments available for selection in the Python node. This list is maintained by the administrator to match the server's Python configuration.
• Allow unknown environments: Controls whether users can enter environment names not listed in Known environments.
  • If false (default on server editions): Strict validation is active. The node will fail if configured with an environment other than PY_ENV or those in the Known environments list.
• Use PY_ENV as (Linux only): Defines the isolation strategy. Passed to python_run.sh via the PY_ENV_USAGE variable. The predefined values are:
  • venv: Treated as a path to a virtual Python environment.
  • image: Treated as a docker image name (one environment per image).
• Container CLI (Linux only): Specifies the containerization tool.
  • Condition: Available only when Use PY_ENV as is set to image.
  • Variable: Passed to python_run.sh via DOCKER_OR_PODMAN.
  • Values: docker or podman.

We've added the parameter Environment variable PY_ENV to the Python node config wizard: It is displayed if the Pass node environment variables setting is true and Python is running in a separate process. The parameter value is set through a drop-down list, which contains the following elements:

• Default: The value set in the Default environment (PY_ENV) component parameter (PY_ENV) is used.
• Values from the Known environments list. If the Allow unknown environments component parameter is disabled, you can only select the value from the list, otherwise you can specify your own value.

For the Interpreter path option, the predefined value python_run.sh has been added to the Linux parameters.

The Python node has a Control variables port added. Through the control variable value, you can dynamically select the Python environment and set the value of the Start wait timeout (ms) option.

In the Python node, it is now possible to add an arbitrary number of output sets. If there is more than one output set, the output column setting page of the wizard changes so that each output table is configured on a separate tab.

Changes to JavaScript and JS Fetch API

Updates to the JavaScript component:

Multiple output datasets: The component now supports a customizable number of output datasets. When multiple datasets are used, the wizard organizes column settings into separate tabs for each output table.

Updates to JS Fetch API:

• SSL verification: HTTPS server certificates are now validated by default (previously disabled).
• Credentials: Added support for requests containing HTTP authentication credentials (username/password).
• Timeouts: Added the AbortController class to enable timeout configuration for Fetch API REST requests.

New functions and preview in calculators

In the Calculator, new functions for determining the number of full time intervals between the dates were added:

• QuartersBetween (Date1, Date2[, Absolute_Value = True]): Returns the total number of quarters between two dates.
• WeeksBetween (Date1, Date2[, Absolute_Value = True]): Returns the total number of weeks between two dates.

If Absolute_Value = False and Date2 is earlier than Date1, these functions will return a negative number of time intervals between the dates.

We have also added some functions of the SHA-2 cryptographic family: SHA256, SHA384, SHA512, SHA512_224, and SHA512_256.

A debug console, similar to the one in JavaScript nodes, has been added to the Preview window for Calculator (Table), Calculator (Variables), and Calculator (Tree).

If an error occurs in expression mode, the console displays the error description and a call stack. The stack is formatted as a comma-separated string of functions and operators, tracing the error from its origin up to the initial call. This trace is also recorded at the debug logging level.

In the descriptions of the Calculator functions, Khi2 is replaced by Chi2. The function names DKhi2, FKhi2, PKhi2, and InvKhi2 remain for backward compatibility, i.e. both variants of these functions are available by name, but only new names are available in the description.

New features when importing data from file sources

Multiple import of text and Excel files can now be executed in parallel processing mode, which speeds up the import process. The corresponding parameters are added to the wizards of these nodes.

We have enhanced text and Excel imports with a new Autodetect columns option. This functionality works similarly to a database select query, allowing you to automatically determine parameters for All columns or New columns only. By enabling Autodetect columns, the node dynamically adapts to structural changes. If a source file arrives with extra columns, the node processes them automatically without needing manual reconfiguration

In XML File import nodes, a customizable property Validate against Schematron has been added. If the option is enabled, the Schematron verification of the rules described in the XSD schema is performed when importing.

Changes to other components and connections

Row Filter: You can now define a list of filter values using a control variable. This feature is supported for fields with integer and string data types.

In Grouping and Table to Variables nodes:

• New separators: Added support for New line Windows (CRLF), New line Linux (LF), and Tabulation (⭾) in the List aggregation option.

In the Imputation component:

• New method: Added Replace with the previous value. This fills data gaps using the preceding non-empty value and supports all data types.

In the Join, Union, and Enrich Data components, the list of required output fields has changed. Previously, all fields in the output sets were required, but now only the following:

• In Join: All fields in the output dataset have become optional.
• In Union and Enrich Data: All fields except for the fields connecting the input datasets are not required.

In Join, we also added the possibility of joining data using a case-independent comparison of keys. By default, a case-sensitive comparison is performed, which keeps to the previously implemented logic. In order for the keys to be compared case-independently, you need to disable the Case-sensitive option in the configuration wizard of the node.

In the Sort node, sorting is now carried out in parallel via several threads. For different use cases, the node processes data from 2 to 7 times faster.

Also, a configurable parameter Maximum thread count, which has a default value Autodetect, was added. Autodetection works as follows:

• If the input data is cached from previous nodes (e.g., Field Features) or the Cache all data option is enabled, the number of threads equals the number of logical processors.
• Otherwise, the number of threads is 1, and a single-thread sorting will be performed.

For optimal operation of the node, we recommended enabling Autodetect for the Maximum thread count parameter.

In components for working with data trees, the root node can now be optional (previously, the root node has always been required, even if it was not necessary). It is important to note that:

• All nodes in the tree need to be optional, because if some node is required, then the entire chain of parent nodes, including the root, is considered required.
• None of the tree nodes should be reconfigured by the user.

Here are the conditions of the root node to be optional in different components, provided that none of the tree nodes has been reconfigured by the user:

• In supernodes : always
• Tree to Table : in the config wizard, no node is selected
• Tree to JSON: always
• Calculator (Tree): no expressions are set
• Union (Tree): always
• Join (Tree): in the configuration wizard, the node of the main tree to which the join is performed is not selected.

In Field Features, the excluded fields are now considered optional. Previously, fields marked as excluded were required. If such a field was absent in the previous node, then activating the node would result in the "Input data source" port not configured error. Such behavior contradicted the logic of excluding unnecessary fields from further processing. In the new version, if the user indicates that the field is excluded, it is not deleted during mapping, but rather, displayed in the input port wizard as Excluded.

The application now provides the ability to copy/paste variables  from one port/node to another.

In the Kafka connection, hostname check is now enabled in the server certificate (ssl.endpoint.identification.algorithm=https). When the Always check all certificates option is enabled in the MGD security settings, it is impossible to redefine the following values: enable.ssl.certificate.verification, ssl.endpoint.identification.algorithm (check of the host), ssl.ca.certificate.stores, ssl.ca.location, and ssl.ca.pem.

The XSD Schema Set connection wizard is now a single page.

New opportunities when working with visualizers

Visualizers can now be transferred from one node to another. Commands Cut, Copy , and Paste were added to the toolbar of the configured visualizers page.

In the Table visualizer, when you select the Listed/Not listed filters, you can now get a list of unique values for the column. (Previously, this functionality was not implemented in this visualizer.)

Updated mapping of tree fields and nodes

In the wizard, we added the ability to exclude fields. In the input field list, use the Add to excluded command (the field should not be a required input or lost). A similar button appeared in the output field list. Use it to add an output field to Excluded list, if the field is not associated with an input field. If the output field is associated with an input field, the command Add to excluded will add both the output field and the associated input field to excluded. Note that it is impossible to add a required or lost input field to the list of excluded fields.

The excluded elements in the list are highlighted in a separate group Excluded. They can be removed and moved. When selecting an excluded field, the input field corresponding to it is highlighted. When performing Auto link , no input fields corresponding to the list of excluded fields will be linked.

Mapping in Megaladata

In the data tree mapping wizards, the ability to exclude nodes has been added. In the list of excluded nodes, you can add a non-connected input node using the context menu command Add to excluded. In the output tree, you can add an Excluded subnode using the Show excluded switch. If there are already some items in the excluded list, you can delete or close this subnode.

Excluded tree elements are highlighted in italics. They can be removed or moved. When selecting an excluded node, the input node corresponding to the full path of the excluded node is highlighted. When performing the Auto link action, the input nodes satisfying the list of excluded are not linked.

Required fields in import nodes

Determining required fields in import nodes: If you select specific fields (e.g., by checking them), they are treated as required outputs of import nodes. These fields must be mapped to input fields, and cannot be excluded in mapping wizards when setting up output ports.

The table describes the required and optional fields for the components responsible for the import of data:

New administration opportunities

Password policy

A new group of settings, Password policy settings, has been added to the administration section of MGD Enterprise. These settings define the minimum requirements for local account passwords and are stored in Settings.cfg.

There are five predefined policy profiles with varying requirements, as well as a customizable user profile.

Regardless of the selected policy, the maximum password length is limited to 1024 characters. Additionally, the existing restriction remains: passwords cannot begin or end with a space.

Please note that password policy restrictions do not apply to existing passwords.

Security parameters

In the security parameters, an option to enforce global SSL certificate verification during the TLS connection stage has been added. This parameter affects file imports by URL, the JavaScript component (Fetch API), LDAP and OpenID authentication, and the following connections:

• REST-service
• SOAP service
• Kafka
• MySQL
• PostgreSQL
• ClickHouse

A similar setting Always check HTTPS certificates includes a forced verification of the server certificate for HTTPS connections and affects the operation of import from file sources by URL, the JavaScript component (Fetch API), OpenID authentication, and connections to web services.

Added settings to limit concurrent sessions (simultaneous connections using the same account from different locations):

• Disable user concurrent sessions: Prevents a user from logging in if there is already an active session for that account (disabled by default).
• Restrict admin concurrent sessions: Configures behavior for Administrator accounts when multiple sessions are attempted. There are three options:
  • No: Administrators can log in with the same account from multiple locations simultaneously (default).
  • Block login: Prevents the login if an active session already exists for the account. This mode is not recommended. If used, ensure that Session disconnection timeout and Session lock timeout are enabled to prevent stale sessions from permanently locking out administrators.
  • Terminate existing sessions: Closes any active sessions for the account before processing the new login. If the existing session cannot be closed, the new login is denied.

When working on Windows, the optional use of OpenSSL instead of SChannel is added to HTTPS requests. To use it, set a true value for the security parameter OpenSSL has higher priority than SChannel for HTTPS. The parameter applies to the REST Service and SOAP Service components. The option can solve the SSL/TLS error problem when performing requests to external systems with activated SSL certificate check.

Connection parameters

For Unix socket connections, a new Unix socket permissions option is available. This option defines which users are authorized to use the socket, thereby securing the data exchange between processes.

The parameter value is specified in octal notation. The default value is 770. MGD Server and MGD Integrator run as different users but share a common group used for communication via the Unix socket.

Since Unix sockets only require write permission, it is unnecessary to grant read or execute permissions. Therefore, a value of 220 is sufficient.

This parameter is only available if MGD Server is running on Linux. MGD Server must be restarted for changes to take effect.

OpenID and LDAP parameters

The Response format option used to generate an authentication request via OpenID and analyze the response after successful authentication is added to OpenID settings. The parameter can have one of two values:

• Request parameters: The response parameters from the OpenID provider are located after the sign ? in the address bar.
• Fragment parameters: The response parameters are located after # (used by default).

The option allows you to configure authentication through Google. Please note that their tokens are not JWT, so any information other than the username is possible to obtain.

You can check the LDAP and OpenID settings in the LDAP Settings and OpenID Settings, respectively, clicking Test in the Authentication testing option.

When testing LDAP authentication, the content of the parameters is checked: Host, Account, Password, Basic Domain, LDAP Filter, and Attribute.

Information about the test connection is displayed in a separate window. In the same window, you can perform a test search for the user.

LDAP

When testing OpenID, a new browser page opens, directing you to the external authentication provider. Upon successful authentication, this page automatically closes, and a new window appears displaying the OpenID test results. These results detail all stages of the authentication process, from the code-token exchange to the retrieval of custom stamp values.

Added Do not check server certificate settings in the LDAP and OpenID settings. Now, the server certificate is checked by default (because of this, LDAP and OpenID authentication may stop working). If the check is not needed, turn it off by enabling the above parameter.

Logging Settings

In the logging parameters, the Log file deletion option is added (available in server editions). If the parameter value is true, the information about the file deletion and File Storage directories is recorded in the log.

Additionally, logging for the number of rows in the output table port has been added. Information about the number of records is written to the node log if the Log row count option is enabled in the node settings, or if Log row count upon node activation is set to true.

User page

In the user account settings, a new setting Allow to save the password to the database has been added. The option is available only for the Workflow design role. It is taken into account when configuring a database connection. If the checkbox is enabled, the value of the Login prompt parameter will be automatically set to true when you open the connection configuration wizard. The checkbox does not affect previously configured connections. The Login prompt checkbox is automatically set only when the connection settings wizard is opened. The value of this parameter can also be obtained from the OpenID token.

When updating Megaladata to version 7.3 (where the Users.cfg file was created in earlier versions), the rights to save passwords for connections are granted to all users with the Workflow design role. If there is no Users.cfg file when you run the server, the user created by default is given the rights to save passwords for connecting to the database.

The Megaladata administrator can now require the user to change their password. In the user editor window, the Password change required option has appeared. If the checkbox is enabled, the user will receive a pop-up window requesting a password change at the next login. After the user changes the password, the request will not appear again. Information about password changes and unsuccessful attempts is recorded in the log.

On the "Users" page, we added search capabilities for the Login and Full name columns:

• The Search button was added to the toolbar to call the search form.
• In the drop-down menu of these columns, a search bar is now available.

Shared folders

The name of a shared folder can not contain characters: \ / : * ? " < > |.

Job scheduler

The Job scheduler now has additional triggers to start jobs:

• Run after completion of another job: Allows you to configure the job to start after a successful or unsuccessful execution of another job, several jobs, or the same job.
• Run on server launch: The reference point to run the package is the moment the server launches.

A job in the Scheduler has a Maximum duration parameter, which is available if the Limit execution time checkbox is marked. The Maximum duration value can be set in seconds, minutes, hours, or days.

When creating a schedule in the Scheduler, you can now specify not only the days of the week or month, but also the start time and duration (for example, run the package from 9:00 every hour for 8 hours). It is also possible to combine days of the week and days of the month when setting up the job launch schedule.

Job Scheduler

In the Variables tab of the job settings window, a Select action has been added, which allows you to select all or several variables from the package to add to the list of job variables. This feature is especially convenient when there are many variables in the package.

In the Scheduler log, we added the ability to view advanced information about errors. When clicking on the error text in the table of completed sessions, a window opens with detailed error information.

Batch Launcher

In Batch Launcher, the ability to connect to MGD Server via a Unix domain socket has been added. The Unix domain socket file name is indicated via the port parameter.

A new start parameter, -deactivate, has been added. It controls the sequential deactivation of node groups after they are fully executed to release memory. Previously, -deactivate=false disabled automatic deactivation (even if enabled in settings), and -deactivate=true enabled it. Now, using the option without a value (i.e., -deactivate) acts as -deactivate=true.

• -.deactivate=value– for a package variable
• -PortName.deactivate=value– for a variable node located inside the port with the name PortName name.

When Batch Launcher starts without parameters, the first output line now displays the program name and the version of Megaladata Batch Launcher, followed by the utility command syntax. (Previously, only the syntax was displayed).

Automatic deactivation of node groups after execution

Parameters have been added that allow you to configure a forced chain deactivation of node groups after they are fully executed, to release the memory used by the workflow. The option can be enabled globally in the server settings, configured at the package level, set in the Job Scheduler, or passed as a Batch Launcher parameter.

Node deactivation is triggered after the node is unlocked during activation. At this point, the system checks whether the node has descendants (nodes connected to the output ports, excluding custom order connections) and whether the node is an output node. In the absence of descendants, and if the node is not an output node, the process of deactivating the nodes in reverse order starts.

This process does not include nodes that are active, or nodes connected to output ports that have not yet been included in the deactivation list. In this case, node deactivation will not compete with activation in parallel workflow branches. The node execution order does not affect the automatic deactivation order; that is, when trying to deactivate a node, all its descendants are also deactivated (except for execution order connections).

The propagation of node deactivation along the chain works via the links, except when they refer to nodes from common packages.

Peer authentication

In Megaladata 7.3, a new type of Peer authentication is available for Linux. This authentication method uses the operating system username and eliminates the need to enter a password for local connections, reducing the risk of losing account credentials.

Peer authentication is only applicable when connecting to MGD Server through a Unix socket. Specifically, this type of authentication is used for the user account under which Integrator or Batch Launcher connects to MGD Server. For users connected to the server interactively through a browser, this option is not applicable.

In the Add/edit user window (in Administration), there is a new System user field, which indicates the UID or username of the current system user. If System user is not specified, it is assumed that the operating system username matches the Login field. The field is available for editing if the Peer type is selected in the Authentication field.

In the configuration files Users.cfg and Integrator.dll.config, a connection is configured for the service user. Thus, with a new installation, Integrator will connect to MGD Server with Peer authentication by default.

Megaladata installation on Linux

When Megaladata is installed on Linux and the Admin user is added, the checkbox requiring a password change is enabled by default. Thus, the administrator password is changed not during installation, but at the first Admin login through the web interface. This option is especially useful for cloud installations.

When updating MGD Server installed as a service, the user version of the server.json file is now preserved. Previously, this file was replaced by the standard version from the distribution, and user settings were lost. This could lead to the inoperability of Megaladata after the update and require additional configuration.

Debian 13 (offered by default) and Astra Linux 1.8 have been added to the list of base images for Megaladata Server.

Now you can use base images not only from docker.io (as before), but also from other repositories.

Performance optimizations

Low-level optimizations have been implemented, allowing for better utilization of hardware resources.

The process for determining executed packages and nodes in the Integrator has been optimized, increasing performance in cases where a large number of Integrator queries are executed. These queries can now be processed in fractions of a second.

The operation of the parallel Loop has been optimized. The output dataset is now populated in parallel using multiple threads. As a result, in some cases, the Loop now performs almost 5 times faster.

Expressions in the Calculator are now calculated 7%–70% faster (depending on expression complexity and functions used).

The following functions have been specifically optimized: Replace, Trim, Upper, Lower, CRC32, MD5, and SHA1.

The creation of package node trees upon opening has been optimized. As a result, large packages now open 20% faster, with significantly reduced RAM consumption.

The Tree mapping wizard now opens faster.

Optimization has affected many parts of the platform. Depending on the usage model, users will experience different increases in speed. Measurements show that the speed boost is observed both when performing simple workflows with typical data operations (import, sorting, grouping, etc.) and when processing large amounts of data in high-load systems.

When running on Windows, the speed is on average 20% higher compared to version 7.2. For Linux, the progress is even more noticeable: the new version is 28% faster.

Usability improvements

For Workflow design and connecting nodes, you can now generate a caption automatically based on the component settings. The auto-generated caption will abbreviate the node settings. For example, for import from file sources, the automatic caption will contain a file name or variable name if the path to the file is set through a variable. For the Column Union node, it will be formed as: Info: + list of tags of information columns + Transp.: + list of tags of transposed columns.

The user can manually choose the mode for generating the node caption. In packages created in version 7.3, the User caption mode will be available for all nodes, but the Automatic caption mode will be used by default.

A copied or cloned node inherits the caption mode. If the original node caption was formed automatically, the copied nodes will not have an index added. For the Connection node, the caption will change if the node is reconfigured for another connection.

For the Connection node, the caption will change if this node is reconfigured for another connection.

The maximum number of rows for node captions is limited. Multi-line captions could completely overlap the node and elements below, blocking access, so only the first three lines are displayed for long captions.

For MGD desktop editions, window titles display the name of the package and the path inside the package.

Markdown syntax can now be used to format the text of annotations and comments. The Use Markdown syntax for annotations and comments option has been added to the package properties and is enabled by default. In annotations, you can also use images and hyperlinks.

In the modal Search panel of Navigation, there is now the ability to search for workflow elements by both GUID and Mark. You can also specify the type of component by which the search will be performed: Node, Derived Component, Connection, etc. The search can be started from the selected node in the Navigation tree by enabling the Start search from selected option.

In the cube visualizer, if you hold Alt, the cube's cross-table will be highlighted in all detailed cube charts (if it does not belong to a hidden series). If you click on a data point in a cube chart, the related cell of the cross-table will be highlighted, and the table will scroll to it.

For package variables, user variables, and control variables, a multi-line Description field has been added. In the Calculator (Variables) node, the Description field is transmitted from the expression to the Description field of the output variable, including in Replace mode.

When cloning a node, its visualizers are transferred to the newly created node.

The Processes pane now displays the remaining time of the process.

Improved automatic node alignment in the workflow construction area.

Improved Error Reporting & Logging

Previously, if a connection error occurred that was unrelated to TLS, the message was uninformative. We have added dedicated processing for such situations so that users receive detailed error information.

Error messages when testing connections have become more informative. The user now receives specific information about the cause of the problem.

When connecting through OpenID, a "Load" indicator is displayed.

If it is not possible to restore a link to a package while opening it, the error message will indicate which package contains the erroneous link.

When MGD Server is stopped, a message recording this event is added to the log.

Information about the MGD version is now added to the beginning of each log file.

All errors and warnings are now displayed in a separate window, where you can also view a detailed description of the error.

The error message displays the time the error occurred (if possible).

Some button captions were changed from “Apply” to:

- “Change”: when editing
- "Add": when added something
- “Save”: if saving occurs at this point
- "OK": in all other cases

In all deletion requests, the text of the "Yes" button is replaced by "Delete".