Version 2.4.0 Released

Apache InLong (应龙) recently released version 2.4.0, which resolved 58 issues, including 10 new features and 45+ optimizations and bug fixes. This version focuses on a comprehensive security hardening of the whole data integration link, covering permission verification, SQL injection defense, command execution and file path validation. It also extends the TransformSDK with ProtoBuf and complex type processing capabilities, optimizes the DataProxy Go SDK and the Python DataProxy SDK release process, and improves the routing management and concurrency safety of the Audit module. Additionally, numerous other features were implemented in Apache InLong 2.4.0.

About Apache InLong

As the industry's first one-stop, all-scenario massive data integration framework, Apache InLong (Yinglong) delivers automated, secure, reliable, and high-performance data transmission capabilities. It enables businesses to rapidly build stream-based data analysis, modeling, and applications. Currently, InLong is widely used across industries such as advertising, payment, social media, gaming, and artificial intelligence, serving thousands of business use cases. It handles over a million billion records/day in high-performance scenarios and over a hundred billion records/day in high-reliability scenarios.

The core positioning of InLong revolves around three keywords: "one-stop," "all-scenario," and "massive data." For "one-stop," InLong aims to shield technical complexities by providing complete data integration and supporting services for out-of-the-box usability. For "all-scenario," it offers comprehensive solutions covering common data integration scenarios in big data ecosystems. For "massive data," its architecture leverages data pipeline layering, fully extensible components, and built-in multi-cluster management to stably support data scales beyond millions of billions of records/day.

Overview of Version 2.4.0

Apache InLong (应龙) recently released version 2.4.0, which resolved 58 issues, including 10 new features and 45+ optimizations and bug fixes. Key enhancements include:

• Security: Comprehensive security hardening across Manager and Agent, including permission verification, SQL injection defense, command whitelist validation and file path validation
• SDK: TransformSDK supports ProtoBuf description files with multiple Proto files and multi-level nested types, and supports encoding of List/Struct/Binary complex types
• SDK: DataProxy Go SDK improves connection initialization, worker selection and error diagnostics; the Python DataProxy SDK now builds standard wheels based on PEP-517
• Audit: Optimized audit route management and resolved thread-safety issues in the Audit SDK under high concurrency

This version also optimizes the operational and maintenance experience for Apache InLong. Other significant features are detailed below.

Dashboard Module

• Added the missing agent installation method on the agent page
• Optimized text display for long forms on the resource details page, improving readability

Manager Module

• Enhanced verification against SQL injection, and fixed the security vulnerabilities in the /api/node/testConnection and /api/cluster/testConnection interfaces
• Added command whitelist validation, and added permission verification for streamsource deletion, datanode modification and tenant role creation/modification
• The default value of openapi.auth.enabled is now true, so OpenAPI authentication is enabled out of the box
• Added parameter validation for AuditAlertRuleRequest order fields and for the node parameter in clusterController
• Fixed the incorrect results returned by the LIKE matching operator when querying MQ messages, and added proper error handling when the MQ message query thread pool is exhausted
• Improved the completeness of the audit query nodeType mapping to avoid null audit IDs

Agent Module

• Fixed the path traversal risk caused by an unvalidated file source path
• Replaced /bin/sh -c with ProcessBuilder to avoid the risk of command injection
• Fixed the missing case in a switch statement

Sort Module

• Sort Format supports outputting complete row information when field parsing errors occur, making troubleshooting easier
• Fixed the NPE thrown by SortConfigUtil.checkUpdate when DataFlowConfig.version is null, which caused SortConfig reload to permanently stall

SDK Module

• TransformSDK supports decoding and transformation of ProtoBuf description files with multiple Proto files and multi-level nested types, and supports parsing map nodes
• TransformSDK supports encoding of List, Struct and Binary type fields for RowData, and adds the concat_struct/extract_struct/extract_binary functions
• TransformSDK supports the "not in" operator and SQL reserved keywords as source field names (backtick-quoted columns)
• Fixed the issue where PbSourceData returned protobuf default values instead of null for unset fields, which caused incorrect data to be written to Iceberg/Parquet sinks
• DataProxy Go SDK adds a connection timeout and tolerates partial endpoint failures during client initialization, optimizes the worker selection logic, supports a retry mechanism on server errors, and logs the remote server address on send timeouts and errors
• Python DataProxy SDK builds standard wheels based on PEP-517, adds a compile Dockerfile, and optimizes dependency version requirements

Audit Module

• Optimized audit route management, and audit routing data source management now supports domain names
• Optimized the auditing data statistics by day, and the day-level API fills with zero when there is no data
• Resolved thread-safety issues in the Audit SDK under high concurrency
• Fixed the ClassLoader leaks and shared state corruption caused by the static ScheduledExecutorService in PulsarSink
• Fixed the audit SQL script initialization failure

Common Module

• Added JSON config support, and fixed the issue where jsonconfig could not be serialized correctly

CI Module

• Fixed the GitHub Action workflow policy violations, upgraded the actions version and specified the permission of actions
• Fixed the failure of all unit tests

Key Features of Version 2.4.0

Comprehensive security hardening of the data integration link

Security is a core capability of a data integration framework. In this version, the community carried out a systematic security review and hardening of Manager and Agent, covering the most common attack surfaces of an open management plane:

• Permission verification: Operator permissions are now verified when deleting a streamsource, modifying a datanode, and creating or modifying tenant roles, preventing unauthorized operations on resources.
• SQL injection defense: The verification against SQL injection is enhanced, and the security vulnerabilities in the /api/node/testConnection and /api/cluster/testConnection interfaces are fixed.
• Command execution: A command whitelist validation is added in Manager, and the Agent replaces /bin/sh -c with ProcessBuilder, eliminating the risk of shell command injection.
• File path validation: The Agent validates the file source path to prevent path traversal.
• Secure defaults: The default value of openapi.auth.enabled is changed to true, so OpenAPI authentication is enabled by default rather than requiring manual configuration.

Contributed by @fuweng11 via INLONG-12129, INLONG-12133, INLONG-12135, INLONG-12144, INLONG-12156, INLONG-12160, and by @spiritxishi via INLONG-12143, INLONG-12148, INLONG-12149.

TransformSDK supports ProtoBuf and complex type processing

InLong Transform adapts to complex and diverse data analysis scenarios on the distribution side. In real-world business, ProtoBuf descriptions are usually split into multiple Proto files and contain deeply nested structures, and the target sinks often require complex types instead of flat columns. This version substantially extends the expressive power of the TransformSDK:

• Decoding and transformation of ProtoBuf description files with multiple Proto files and multi-level nested types, and parsing of map nodes.
• Encoding of List, Struct and Binary type fields for RowData, so that complex types can be written to sinks such as Iceberg and Parquet directly.
• New concat_struct, extract_struct and extract_binary functions for constructing and extracting complex types.
• Enhanced protobuf processing and SQL alias parsing, support for the "not in" operator, and support for SQL reserved keywords as source field names via backtick-quoted columns.

Contributed by @luchunliang via INLONG-12108, INLONG-12111, INLONG-12113, INLONG-12117, INLONG-12125, INLONG-12158, INLONG-12056.

DataProxy SDK optimization for Go and Python

The DataProxy SDK is the entry point for business data reporting, so its robustness and ease of integration directly affect the reporting success rate. This version improves both the Go and the Python SDK:

• The Go SDK adds a connection timeout and tolerates partial endpoint failures during client initialization, so a few unavailable DataProxy nodes no longer block the client from starting.
• The Go SDK optimizes the worker selection logic, supports a retry mechanism when a server error occurs, and logs the remote server address on send timeouts and server errors for faster troubleshooting.
• The Python SDK builds standard wheels based on PEP-517 and provides a compile Dockerfile, making the release and installation process reproducible.

Contributed by @yfsn666 via INLONG-12119, INLONG-12120, INLONG-12121, INLONG-12058, INLONG-12062, INLONG-12016, INLONG-12054, and by @hzqmwne via INLONG-12090.

Audit route management and concurrency safety optimization

Following the horizontal scaling capability of Audit Store introduced in 2.3.0, this version continues to improve the usability and stability of the audit link:

• Optimized audit route management, and audit routing data source management supports domain names, so that storage clusters can be configured without hard-coded IP addresses.
• Optimized the auditing data statistics by day, and the day-level API fills with zero when there is no data, keeping the trend curves continuous.
• Resolved thread-safety issues in the Audit SDK under high concurrency, improving the accuracy of audit data.

Contributed by @doleyzi via INLONG-12060, INLONG-12088, INLONG-12067, INLONG-12137.

Future Plans

In version 2.4.0, we have systematically strengthened security and enriched the data processing capabilities of the SDK. Welcome everyone to use it. If you have more scenarios and requirements, or encounter any problems during use, please feel free to raise issues and PR. In future versions, the InLong community will continue to:

• Support for collecting from more data sources
• Real-time synchronization supports more data sources and data targets
• Continuously improve the security and permission management of the whole link

We welcome contributions from developers interested in InLong!