12 KiB
Example: Runtime Architecture Migration in a Data Pipeline (Software Engineering)
Status: Draft Phase: The Bedrock Phase
What This Example Demonstrates
Context record structure (OE-0003) applied to a programming language and runtime architecture selection, illustrating how the reasoning behind a Python-to-Rust migration is typically lost when only the new codebase is preserved (OE-0009, OE-0010).
The Observation
A data ingestion pipeline originally written in Python processed 40,000 events per second using a synchronous, single-threaded design with a PostgreSQL persistence layer. As event volume grew to 400,000 events per second, the Python implementation exhibited increasing latency spikes (p99 rising from 45ms to 2.3 seconds) and garbage collection pauses that caused the process to miss its processing window, resulting in backpressure buildup and eventual queue overflow. Profiling revealed that 78% of CPU time was spent on object allocation and deallocation in the event deserialization path, and the Global Interpreter Lock prevented effective parallelization of the CPU-bound parsing work.
A developer unfamiliar with the system's history might look at the Rust replacement and ask: why was Python ever used for a high-throughput pipeline? The answer is that the pipeline was never designed to be high-throughput. When it was built two years earlier, the event volume was 5,000 events per second, and Python's ecosystem (existing database libraries, rapid prototyping speed, team familiarity) made it the correct choice. The context that is almost never preserved is the reasoning chain: what was true when Python was selected, what changed, why Rust became correct later, and what the new system inherits from the old one.
Engineering Translation
This example demonstrates a pattern that is pervasive in software engineering: the reasoning behind an initial architecture decision is lost when the architecture is replaced, because the replacement is documented as a new system rather than as an extension of a prior decision. A developer who joins the team after the Rust migration sees a high-performance concurrent pipeline and may assume the project always required that level of performance. Without the context record, they cannot reconstruct why the original Python system existed, what conditions triggered the migration, or what constraints from the Python era still inform the Rust design (data schemas, queue contracts, error handling semantics, monitoring expectations).
The context loss has practical consequences. If a future developer proposes migrating a different pipeline from Python to Rust based on this project's example, they need to understand whether the conditions that justified the migration here (CPU-bound parsing, GIL limitations, predictable data schemas) apply to their pipeline. Without the context record, the Rust pipeline becomes a template to copy rather than a decision to evaluate — which is the opposite of what Open Engineer intends.
Context Record
| Field | Content |
|---|---|
| Decision | Migrate the event ingestion pipeline from Python (synchronous, single-threaded) to Rust (asynchronous, multi-threaded) while preserving the existing PostgreSQL persistence schema, message queue contracts, and monitoring instrumentation. |
| Observation | (1) Production metrics over 6 months showed event volume growing from 40,000 to 400,000 events/second with p99 processing latency rising from 45ms to 2.3 seconds. (2) Profiling with py-spy showed 78% of CPU time in object allocation/deallocation during event deserialization, with GC pause frequency increasing linearly with event rate. (3) The Global Interpreter Lock prevented distributing parsing work across the available 16 CPU cores — multi-process worker pools were tested but introduced inter-process serialization overhead that eliminated the throughput gain. (4) Load testing a prototype Rust implementation using tokio showed 620,000 events/second with p99 latency of 3.2ms on the same hardware, with memory allocation remaining flat under sustained load due to Rust's ownership model eliminating GC pauses. |
| Alternatives | (A) Optimize the Python implementation with Cython or PyO3 extensions for the parsing hot path — rejected: profiling showed the bottleneck was spread across multiple allocation sites in the deserialization logic, not concentrated in a single function that could be offloaded to an extension. The engineering effort to rewrite the allocation pattern in Cython was estimated at 60% of the full Rust migration cost while delivering at most 3x throughput improvement, insufficient for the 10x target. (B) Switch to a compiled language with garbage collection (Go) — rejected: Go's garbage collector, while lower-latency than Python's, still introduces stop-the-world pauses in the microsecond range that compound at the target throughput. The Rust prototype's zero-cost abstraction model (no GC, no runtime) demonstrated that the GC elimination itself was responsible for approximately 40% of the latency improvement. (C) Horizontal scaling with more Python instances behind a load balancer — rejected: the queue-based architecture already distributed work, but the per-instance throughput ceiling (approximately 60,000 events/second) meant that scaling to 400,000 events/second required 7+ instances, each consuming 4GB of memory, for a total memory footprint of 28GB. The single Rust instance achieved the same throughput in 800MB. The operational complexity of managing 7+ Python processes (separate health checks, separate log aggregation, separate deployment coordination) was disproportionate to the benefit. (D) Rust — selected. |
| Constraints | Must preserve the existing PostgreSQL schema (downstream consumers depend on the current table structure and column types). Must maintain backward compatibility with the existing message queue protocol (events published by upstream producers cannot be modified). Must not introduce a new runtime dependency (no JVM, no Erlang VM) — the operations team supports Python and compiled binaries only. Migration must complete within a single quarter (3 months) because the current system's p99 latency is approaching the SLA breach threshold at the projected 6-month growth rate. The Rust implementation must run on the existing Linux deployment infrastructure without kernel-level modifications. |
| Reasoning | The migration is driven not by a preference for Rust but by a specific, observed failure mode: Python's runtime characteristics (GC pauses, GIL, object allocation overhead) create a throughput ceiling that the system has reached. The Rust prototype demonstrated that eliminating GC and enabling true multi-threaded parsing removes this ceiling. The decision preserves continuity by maintaining the data schemas, queue contracts, and monitoring infrastructure from the Python system — the Rust system inherits the interface obligations of the Python system and extends them with higher throughput. The trade-off is reduced prototyping velocity: future pipeline changes in Rust require compile-time verification and explicit memory management that Python would handle at runtime. This trade-off is acceptable because the pipeline's interface contracts are now stable (2 years of production use) and the primary engineering need is throughput, not rapid iteration. |
| Verification | (1) The Rust implementation was deployed alongside the Python system in a shadow traffic configuration for 72 hours, processing the same live event stream. The Rust system processed 100% of events with zero data loss; a diff of the PostgreSQL output showed byte-identical rows for both systems across 38 million events. (2) Load testing at 1.5x projected peak volume (930,000 events/second) showed p99 latency of 8.1ms, well within the 100ms SLA. (3) Memory profiling under sustained load showed stable allocation at 812MB with no growth trend over 48 hours, confirming the absence of memory leaks that would require process restarts. (4) The operations team validated that the compiled binary deployed and restarted within the existing infrastructure automation without modifications. |
| Lineage | Builds on the original Python pipeline implementation (context record CR-SW-2021-004, "Event ingestion service initial implementation"). That record documented the decision to use Python based on the then-current event volume (5,000/second), team composition (3 Python developers, 0 Rust developers), and delivery timeline (2 weeks to production). The present record extends that lineage by documenting the conditions under which the original decision is no longer valid and the reasoning behind the new architecture selection. The PostgreSQL schema and message queue contracts defined in CR-SW-2021-004 are inherited without modification. |
| Assumptions | The event volume growth rate (approximately 2x per 6 months) will continue at approximately the same rate for the next 18 months, justifying the headroom provided by the Rust implementation. The Rust team's expertise (2 developers at the time of migration) is sufficient to maintain the system without requiring external contractors. The data schemas and queue contracts inherited from the Python system will remain stable — if upstream producers change the event format, the Rust system will require the same adaptation work as the Python system would have. The compiled binary's platform-specific behavior (memory alignment, SIMD optimization) does not introduce subtle differences in numeric processing that would cause the Rust system to produce different results than the Python system for identical inputs — this was verified by the shadow traffic test but only for the data patterns present during the 72-hour test window. |
| Open Questions | At what event volume does the PostgreSQL persistence layer become the next bottleneck, and should the schema be redesigned before that point? Should the team adopt Rust for new pipelines proactively, or continue using Python for new pipelines until they hit the same throughput ceiling? |
What Context Loss Looks Like Here
Without this context record, a developer encountering the Rust pipeline for the first time sees a high-performance system and may reasonably ask: "Why was this ever written in Python?" The answer — that Python was correct for the original constraints and the constraints changed — is lost without the record. Worse, without the record, the team may repeat the same pattern on future projects: starting in Python for rapid prototyping, hitting the same throughput ceiling, and migrating to Rust without ever documenting why the initial choice was correct or what the trigger conditions for migration are. Each cycle repeats the same context loss. The context record breaks the cycle by preserving the reasoning chain from Python-correct to Rust-correct (OE-0009, OE-0010).
Self-Fading Assessment
This example builds a bridge from the abstract concept of context preservation to a concrete software engineering case that most developers have experienced: a system that was right when it was built, became wrong as conditions changed, and was replaced without preserving the reasoning for either the original choice or the replacement. The reader has crossed this bridge when they can look at any architecture migration in their own work — language change, framework change, paradigm change — and immediately identify what reasoning is being lost if the migration is documented only as a new system rather than as an extension of a prior decision. Once that pattern is recognized, the specific details of Python, Rust, and event pipelines become incidental — the underlying principle that every architecture decision exists on a timeline and its reasoning must be preserved across transitions applies equally to all software engineering contexts.