subsystem-summary-of-rust: Скилл для краткого анализа

--- name: subsystem-summary-of-rust description: "read this skill for a token-efficient summary of the rust subsystem" --- # Rust Subsystem (Non-Soroban) — Technical Summary ## Overview The rust subsystem provides a Rust static library (`rust_stellar_core`) that is linked into the stellar-core C++ binary. It uses the `cxx` crate (v1.0.97) to define a bidirectional FFI bridge between C++ and Rust. The subsystem's primary responsibilities are: 1. **Soroban host function invocation** — dispatching to the correct protocol-versioned soroban host. 2. **Fee computation** — transaction resource fees, rent fees, and rent write fees. 3. **Module caching** — pre-compiled WASM module cache for Soroban contracts. 4. **128-bit integer arithmetic** — exposing Rust's native `i128` to C++. 5. **Base64 encoding/decoding** — used for XDR serialization interop. 6. **Ed25519 signature verification** — using `ed25519-dalek` for faster verification. 7. **Logging bridge** — routing Rust `log` crate output to the C++ spdlog system. 8. **Quorum intersection checking** — using the `stellar-quorum-analyzer` SAT solver. 9. **Utility functions** — rustc version, executable path, backtrace capture, XDR version checks. The crate is built as `crate-type = ["staticlib"]` (edition 2021, rust-version 1.82.0). Optional features include `tracy` (profiling), `next` (pre-release protocol), `testutils` (test-only code), and `unified` (IDE-friendly single cargo build). ## File Layout | File | Role | |------|------| | `Cargo.toml` | Crate metadata, multi-host soroban dependencies, feature flags | | `src/lib.rs` | Crate root; declares modules, re-exports bridge symbols, defines `tracy_span!` macro | | `src/bridge.rs` | `#[cxx::bridge]` module — all FFI type/function declarations | | `src/common.rs` | `RustBuf`/`CxxBuf`/`BridgeError` impls; `get_rustc_version`, `current_exe`, `capture_cxx_backtrace`, `check_xdr_version_identities` | | `src/b64.rs` | `to_base64` / `from_base64` | | `src/ed25519_verify.rs` | `verify_ed25519_signature_dalek` (unsafe raw-pointer FFI) | | `src/i128.rs` | `i128_add`, `i128_sub`, overflow/underflow checks, conversion | | `src/log.rs` | `StellarLogger` implementing `log::Log`, routes to C++ spdlog | | `src/quorum_checker.rs` | `network_enjoys_quorum_intersection` wrapping `stellar-quorum-analyzer` | | `src/soroban_invoke.rs` | `invoke_host_function`, fee computation, transaction parsing dispatchers | | `src/soroban_module_cache.rs` | `SorobanModuleCache` struct; per-protocol caches | | `src/soroban_proto_all.rs` | Protocol-versioned host modules (p21–p26), dispatch table, adaptors | | `src/soroban_proto_any.rs` | Protocol-agnostic host invocation code, mounted inside each pN module | | `CppShims.h` | Thin C++ shim functions (`shim_isLogLevelAtLeast`, `shim_logAtPartitionAndLevel`) | | `RustBridge.h` | cxx-generated C++ header with all bridge types and function declarations | | `RustBridge.cpp` | cxx-generated C++ implementation (extern "C" thunks, Vec/Box specializations) | | `RustVecXdrMarshal.h` | Declares `rust::Vec<uint8_t>` as valid xdrpp byte buffer type | ## The CXX Bridge Mechanism ### How it works The bridge is defined in `src/bridge.rs` inside a `#[cxx::bridge]` attribute macro on `mod rust_bridge`. This module contains three sections: 1. **Shared types** — structs and enums visible to both sides, defined once: - `CxxBuf` (C++→Rust data: wraps `UniquePtr<CxxVector<u8>>`) - `RustBuf` (Rust→C++ data: wraps `Vec<u8>`) - `XDRFileHash`, `InvokeHostFunctionOutput`, `CxxLedgerInfo`, `CxxTransactionResources`, `CxxFeeConfiguration`, `CxxLedgerEntryRentChange`, `CxxRentFeeConfiguration`, `CxxRentWriteFeeConfiguration`, `CxxI128`, `FeePair`, `SorobanVersionInfo` - Enums: `LogLevel` (shared with `stellar::LogLevel`), `BridgeError`, `QuorumCheckerStatus` - `QuorumSplit`, `QuorumCheckerResource` 2. **`extern "Rust"` block** (`#[namespace = "stellar::rust_bridge"]`) — Rust functions callable from C++: - All functions listed in the "Key Functions" section below. - The opaque type `SorobanModuleCache` with its methods. 3. **`extern "C++"` block** (`#[namespace = "stellar"]`) — C++ functions callable from Rust: - `shim_isLogLevelAtLeast(partition: &CxxString, level: LogLevel) -> Result<bool>` - `shim_logAtPartitionAndLevel(partition: &CxxString, level: LogLevel, msg: &CxxString) -> Result<()>` ### Data passing convention - **C++ → Rust**: Data is passed as `CxxBuf` containing `UniquePtr<CxxVector<u8>>` (a C++-allocated `std::vector<uint8_t>`). The Rust side reads from it via `data.as_slice()`. - **Rust → C++**: Data is returned as `RustBuf` containing `Vec<u8>` (Rust-allocated). The C++ side reads from `data` (a `rust::Vec<uint8_t>`). - XDR serialization/deserialization is done with `ReadXdr`/`WriteXdr` using `non_metered_xdr_from_cxx_buf` and `non_metered_xdr_to_rust_buf` helper functions with a depth limit of 1000 and length limit matching the buffer size. - `RustVecXdrMarshal.h` allows xdrpp to directly unmarshal from `rust::Vec<uint8_t>`. ### Generated files `RustBridge.h` and `RustBridge.cpp` are generated by the `cxxbridge` tool. They contain: - Full implementations of `rust::String`, `rust::Slice<T>`, `rust::Box<T>`, `rust::Vec<T>`, `rust::Opaque`, `rust::Error`. - C struct definitions mirroring the shared types. - `static_assert` checks ensuring `LogLevel` enum values match between C++ and Rust. - `extern "C"` function declarations for the mangled bridge symbols. - C++ wrapper functions in `namespace stellar::rust_bridge` that call through extern "C" thunks and translate Rust errors to C++ exceptions (`rust::Error`). - Template specializations for `rust::Vec<RustBuf>`, `rust::Vec<XDRFileHash>`, `rust::Vec<CxxBuf>`, etc. - `rust::Box<SorobanModuleCache>` alloc/dealloc/drop specializations. ### CppShims.h Provides simple inline wrapper functions that cxx.rs can call, bridging to C++ APIs that are too complex for cxx to handle directly (e.g., static member functions): - `shim_isLogLevelAtLeast` → `Logging::isLogLevelAtLeast` - `shim_logAtPartitionAndLevel` → `Logging::logAtPartitionAndLevel` ## Key Data Structures ### `CxxBuf` / `RustBuf` Directional byte-buffer wrappers for passing XDR-serialized data across the FFI boundary. `CxxBuf` owns a `std::unique_ptr<std::vector<uint8_t>>` (C++ allocated). `RustBuf` owns a `Vec<u8>` (Rust allocated). Both implement `AsRef<[u8]>`. ### `CxxI128` Split representation of 128-bit integer: `{ hi: i64, lo: u64 }`. Used because C++ lacks native `i128` on all platforms. Converted to/from Rust `i128` via `int128_helpers::{i128_from_pieces, i128_hi, i128_lo}`. ### `InvokeHostFunctionOutput` Return value of `invoke_host_function`. Contains: - `success: bool`, `is_internal_error: bool` - `diagnostic_events: Vec<RustBuf>` (XDR-encoded `DiagnosticEvent`) - `cpu_insns`, `mem_bytes`, `time_nsecs` (and excluding-VM-instantiation variants) - `result_value: RustBuf`, `contract_events: Vec<RustBuf>`, `modified_ledger_entries: Vec<RustBuf>`, `rent_fee: i64` ### `SorobanModuleCache` An opaque Rust type exposed to C++ via `rust::Box<SorobanModuleCache>`. Holds per-protocol `ProtocolSpecificModuleCache` instances (p23, p24, p25, and optionally p26 with `next` feature). Each `ProtocolSpecificModuleCache` contains a `ModuleCache` (from soroban-env-host, threadsafe via internal locking) and an `AtomicU64` tracking memory consumption. Methods: - `compile(&mut self, ledger_protocol: u32, wasm: &[u8])` — parse and cache a WASM module for the given protocol. - `shallow_clone(&self) -> Box<SorobanModuleCache>` — clone shared ownership handles for multithreaded compilation. - `evict_contract_code(&mut self, key: &[u8])` — remove a module from all protocol caches by 32-byte hash. - `clear(&mut self)` — clear all protocol caches. - `contains_module(&self, protocol: u32, key: &[u8]) -> bool` - `get_mem_bytes_consumed(&self, protocol: u32) -> u64` ### `HostModule` A dispatch table struct (not crossing FFI) containing function pointers for a specific protocol version's soroban host. Fields include `max_proto`, `invoke_host_function`, `compute_transaction_resource_fee`, `compute_rent_fee`, `compute_rent_write_fee_per_1kb`, `contract_code_memory_size_for_rent`, `can_parse_transaction`, and `get_soroban_version_info`. The static array `HOST_MODULES` holds one entry per protocol version (p21–p25/p26), populated via the `proto_versioned_functions_for_module!` macro. ### `ProtocolSpecificModuleCache` Per-protocol cache wrapper (defined in `soroban_proto_any.rs`). Wraps a `ModuleCache` from the protocol's soroban-env-host and a `CoreCompilationContext` (unlimited budget for compilation). Supports `compile`, `evict`, `clear`, `contains_module`, `get_mem_bytes_consumed`, and `shallow_clone`. ### `CoreCompilationContext` Implements `CompilationContext` (= `ErrorHandler + AsBudget`) with an unlimited budget, used for compiling WASM modules outside of transaction execution. ## Key Functions (Exported Rust → C++) ### Soroban Host Invocation - `invoke_host_function(config_max_protocol: u32, enable_diagnostics: bool, instruction_limit: u32, hf_buf: &CxxBuf, resources: CxxBuf, restored_rw_entry_indices: &Vec<u32>, source_account: &CxxBuf, auth_entries: &Vec<CxxBuf>, ledger_info: CxxLedgerInfo, ledger_entries: &Vec<CxxBuf>, ttl_entries: &Vec<CxxBuf>, base_prng_seed: &CxxBuf, rent_fee_configuration: CxxRentFeeConfiguration, module_cache: &SorobanModuleCache) -> Result<InvokeHostFunctionOutput>` — Dispatches to the correct protocol-versioned host via `get_host_module_for_protocol`. Wraps the call in `panic::catch_unwind`. ### Fee Computation - `compute_transaction_resource_fee(config_max_protocol: u32, protocol_version: u32, tx_resources: CxxTransactionResources, fee_config: CxxFeeConfiguration) -> Result<FeePair>` — Returns `(non_refundable_fee, refundable_fee)`. - `compute_rent_fee(config_max_protocol: u32, protocol_version: u32, changed_entries: &Vec<CxxLedgerEntryRentChange>, fee_config: CxxRentFeeConfiguration, current_ledger_seq: u32) -> Result<i64>` - `compute_rent_write_fee_per_1kb(config_max_protocol: u32, protocol_version: u32, bucket_list_size: i64, fee_config: CxxRentWriteFeeConfiguration) -> Result<i64>` - `contract_code_memory_size_for_rent(config_max_protocol: u32, protocol_version: u32, contract_code_entry: &CxxBuf, cpu_cost_params: &CxxBuf, mem_cost_params: &CxxBuf) -> Result<u32>` — Only valid for protocol ≥ 23. ### Transaction Parsing - `can_parse_transaction(config_max_protocol: u32, protocol_version: u32, xdr: &CxxBuf, depth_limit: u32) -> Result<bool>` — Checks if a `TransactionEnvelope` XDR can be deserialized in the given protocol. ### 128-bit Integer Arithmetic - `i128_add(lhs: &CxxI128, rhs: &CxxI128) -> Result<CxxI128>` - `i128_sub(lhs: &CxxI128, rhs: &CxxI128) -> Result<CxxI128>` - `i128_add_will_overflow(lhs: &CxxI128, rhs: &CxxI128) -> Result<bool>` - `i128_sub_will_underflow(lhs: &CxxI128, rhs: &CxxI128) -> Result<bool>` - `i128_from_i64(val: i64) -> Result<CxxI128>` - `i128_is_negative(val: &CxxI128) -> Result<bool>` - `i128_i64_eq(lhs: &CxxI128, rhs: i64) -> Result<bool>` ### Ed25519 Verification - `verify_ed25519_signature_dalek(public_key_ptr: *const u8, signature_ptr: *const u8, message_ptr: *const u8, message_len: usize) -> bool` — Unsafe raw-pointer interface. Uses `ed25519-dalek`'s `verify_strict` (rejects small-order points, matching libsodium). Never panics; returns false for invalid input. ### Base64 - `to_base64(b: &CxxVector<u8>, s: Pin<&mut CxxString>)` — Encode bytes to base64. - `from_base64(s: &CxxString, b: Pin<&mut CxxVector<u8>>)` — Decode base64 with error-tolerant stripping of invalid characters. ### Logging - `init_logging(maxLevel: LogLevel) -> Result<()>` — Initializes the `StellarLogger` as the global Rust logger, routing to C++ spdlog. Uses `AtomicBool` for one-time initialization. Log partitions (e.g., `TX`, `Ledger`, `SCP`) are defined in `log::partition` and must match `util/LogPartitions.def` on the C++ side. ### Quorum Checker - `network_enjoys_quorum_intersection(nodes: &Vec<CxxBuf>, quorum_set: &Vec<CxxBuf>, potential_split: &mut QuorumSplit, resource_limit: &QuorumCheckerResource, resource_usage: &mut QuorumCheckerResource) -> Result<QuorumCheckerStatus>` — Returns `UNSAT` (quorum intersection holds), `SAT` (split found, populates `potential_split`), or `UNKNOWN`. Time limit enforced internally; memory limit is a hard abort via global allocator. ### Module Cache - `new_module_cache() -> Result<Box<SorobanModuleCache>>` - Methods on `SorobanModuleCache`: `compile`, `shallow_clone`, `evict_contract_code`, `clear`, `contains_module`, `get_mem_bytes_consumed`. ### Utility - `get_rustc_version() -> String` - `current_exe() -> Result<String>` - `capture_cxx_backtrace() -> String` — Uses `backtrace` crate; filters out initial Rust frames and libc frames. - `get_soroban_version_info(core_max_proto: u32) -> Vec<SorobanVersionInfo>` — Returns version info for all linked soroban hosts. Panics if no host supports the given protocol. - `check_sensible_soroban_config_for_protocol(core_max_proto: u32)` — Validates HOST_MODULES are in ascending order and cover the max protocol. - `check_xdr_version_identities() -> Result<()>` — Compares XDR file SHA256 hashes across crates. ## Multi-Protocol Soroban Host Architecture ### Design stellar-core links multiple versions of `soroban-env-host` simultaneously, one per protocol version range. Each is labeled by its maximum supported protocol (e.g., `soroban-env-host-p21` supports protocols up to 21). At runtime, `get_host_module_for_protocol(config_max_proto, ledger_protocol)` selects the appropriate host. ### Implementation pattern `soroban_proto_all.rs` defines adaptor modules `p21`, `p22`, `p23`, `p24`, `p25`, and conditionally `p26` (behind `next` feature). Each adaptor: 1. Imports its specific `soroban_env_host_pNN` crate and re-exports it as `soroban_env_host`. 2. Provides adapter functions for API differences between host versions (e.g., different field names in `TransactionResources`, `RentFeeConfiguration`). 3. Mounts `soroban_proto_any.rs` as a child module — this file is the same source but "sees" a different `super::soroban_env_host` in each context. 4. Defines stub types (`ModuleCache`, `ErrorHandler`, `CompilationContext`) for older protocols (p21, p22) that don't support the reusable module cache API. ### Protocol dispatch The `HOST_MODULES` static array maps protocol ranges to `HostModule` structs containing function pointers. `get_host_module_for_protocol` iterates this array: each entry's implied minimum protocol is one more than the previous entry's `max_proto` (first entry starts at 0). ### Aliases - `soroban_curr` — alias for the latest non-next host (p25, or p26 with `next`). - `protocol_agnostic` — re-exports from p24 that are stable across versions (e.g., `int128_helpers`, `make_error`). ## Key Data Flows ### C++ → Soroban Invocation → C++ 1. C++ constructs `CxxBuf` objects containing XDR-serialized data (host function, resources, ledger entries, etc.) and a `CxxLedgerInfo`. 2. Calls `stellar::rust_bridge::invoke_host_function(...)` which crosses the FFI boundary. 3. Rust dispatches to the correct `HostModule` based on `(config_max_protocol, ledger_info.protocol_version)`. 4. The protocol-specific `invoke_host_function` in `soroban_proto_any.rs` deserializes XDR, creates a `Budget`, optional trace hook, and calls through to `soroban_env_host::e2e_invoke::invoke_host_function`. 5. Results are re-serialized to `RustBuf` vectors and returned as `InvokeHostFunctionOutput`. 6. The C++ wrapper in `RustBridge.cpp` unwraps the result or throws `rust::Error` on failure. ### Logging (Rust → C++) 1. Rust code calls `log::info!()` etc. 2. `StellarLogger::log()` converts the level and calls `shim_logAtPartitionAndLevel` via the extern "C++" bridge. 3. The shim calls `Logging::logAtPartitionAndLevel` in C++. ### Module Cache Lifecycle 1. C++ calls `new_module_cache()` to get a `rust::Box<SorobanModuleCache>`. 2. Calls `compile(protocol, wasm_bytes)` to cache WASM modules (typically on startup and during catchup). 3. The cache is passed by reference to `invoke_host_function`. 4. `shallow_clone()` creates shared-ownership handles for multithreaded use. 5. `evict_contract_code(key)` removes entries; `clear()` empties all caches. ## Error Handling - All fallible Rust functions return `Result<T, Box<dyn std::error::Error>>` (or `Result<T, HostError>`). - cxx converts Rust `Err` returns into C++ `rust::Error` exceptions. - `invoke_host_function` and `network_enjoys_quorum_intersection` additionally wrap their core logic in `panic::catch_unwind` to convert Rust panics into errors rather than unwinding across the FFI boundary. - The quorum checker's memory limit is a hard abort (non-catchable) by design. - `CoreHostError` enum wraps either a `HostError` from soroban or a general `String` message. ## Dependencies | Crate | Purpose | |-------|---------| | `cxx` 1.0.97 | C++/Rust FFI bridge framework | | `base64` 0.13.1 | Base64 encode/decode | | `log` 0.4.19 | Rust logging facade | | `ed25519-dalek` 2.1.1 | Ed25519 signature verification | | `itertools` 0.10.5 | Iterator utilities | | `backtrace` 0.3.76 | C++ backtrace capture (with `cpp_demangle`) | | `rand` 0.8.5 | RNG (must match soroban's version) | | `rustc-simple-version` 0.1.0 | Compile-time rustc version string | | `tracy-client` 0.17.0 | Tracy profiling (optional) | | `stellar-quorum-analyzer` | SAT-based quorum intersection checking | | `soroban-env-host-pNN` | Protocol-specific Soroban hosts (p21–p26) | | `soroban-test-wasms` | Pre-compiled test WASM binaries | | `soroban-synth-wasm` | Random WASM generation for testing | ## Build Notes - The default build does **not** use the optional `soroban-env-host-pNN` deps from Cargo.toml. Instead, each host is built as a separate cargo invocation and linked in (see `src/Makefile.am`). This avoids Cargo's dependency unification. - The `unified` feature enables all hosts as direct dependencies for IDE usage. This perturbs `Cargo.lock` — changes should not be committed. - Tracy feature flags must match between the Rust crate and the C++ `lib/tracy` submodule version.