Data movement and middleware integration: connecting systems that don’t talk

Almost no business in 2026 runs on a single system. An accounting platform was selected in 2014. The CRM came in later as a replacement for a homegrown contact database that staff outgrew. Inventory logic still lives in a custom Access database dating to 2008. The new cloud SaaS platform the sales team adopted last year does not talk to any of them. By the time the IT director inherits the portfolio, the integrations between these systems are a combination of manual exports, scheduled scripts that no one fully documents, and a no-code tool that handles two of the eight flows the business actually needs. The integration problem is not theoretical. It is the operational vulnerability that consumes more IT staff time than any other category of work.¹

Phoenix Consultants Group has built data movement and middleware integration solutions across more than 500 production engagements since 1995, with hundreds of data transfers documented at zero data loss. This article describes the four integration patterns IT directors encounter most often, when each pattern fits, how PCG handles legacy systems without APIs, and how integrations are validated before going live. The framing is technical. Its audience is the IT director or integration architect responsible for the systems that have to talk to each other regardless of how unfriendly the source systems are.²

Why do business systems need middleware in the first place?

The need arrives at the point where two or more systems hold related data and the business cannot tolerate the manual reconciliation between them. An accounting platform holds the invoice. CRM holds the customer record. Fulfillment system holds the shipping status. When a customer calls about an order, the staff member needs all three. Without integration, that staff member opens three browser tabs, looks up the customer in each system, and reconciles the information mentally on the call. Multiplied across hundreds of calls per week, the manual reconciliation becomes the operational cost the business is trying to eliminate.

Middleware is the architectural answer to that cost. Instead of every system needing to know about every other system directly, middleware sits between them and handles the translation, transport, and reliability concerns. The accounting platform talks to the middleware. CRM does the same. So does the fulfillment system. When a new system joins the portfolio, it connects to the middleware once, not to each of the existing systems individually. That decoupling is what makes integration portfolios maintainable as the business grows past three or four systems.³

Without middleware, integrations are typically built as point-to-point connections: system A talks directly to system B. That works for the first integration. It becomes unmaintainable by the time the business has five systems and ten point-to-point connections, each one with its own error handling, format conversion, retry logic, and ownership. Middleware replaces that mesh with a hub model where each system has one connection to maintain rather than several.

What are the four primary patterns for moving data between systems?

Most integrations PCG builds use one of four patterns, each fitting a specific combination of data volume, latency requirement, and source system capability. The choice between patterns is made during the audit phase, not based on a default preference. Some integrations use a combination of patterns when different parts of the data flow have different characteristics.

Most production portfolios combine these patterns rather than running on a single one. A typical mid-sized business might use real-time APIs between the e-commerce platform and the order management system, scheduled ETL to refresh the data warehouse overnight, message queues to broadcast customer updates to the marketing platform and fulfillment system simultaneously, and file-based exchange for monthly statements going to the accounting partner. The middleware layer handles all four patterns through a consistent architecture, so the IT team operates one integration platform rather than four.²

When does ETL beat real-time integration?

Real-time integration is the architecturally fashionable choice in 2026, and it produces a system that is less expensive to operate when the data flow actually requires real-time behavior. When the data flow does not require real-time behavior, real-time integration introduces complexity and operational burden that batch ETL would have avoided. The decision between the two is operational, not aspirational.

The wrong choice in either direction has measurable cost. Real-time integration where batch would have sufficed produces operational alerts, monitoring overhead, and brittleness that batch would have avoided. Batch where real-time was required produces customer-facing latency and reconciliation work that real-time would have eliminated.

How does PCG handle integrations with legacy systems that have no APIs?

Legacy systems without APIs are the most common integration target PCG encounters in 2026. Modern integration tools assume both endpoints expose REST APIs with JSON payloads. The operational reality is that the system holding the business-critical data is often a Visual FoxPro database from 2003, a Microsoft Access application from 2008, or a mainframe extract program written before the current IT team was hired. Each one requires a different integration method.⁴

Direct database access is the first approach when the legacy system stores its data in a database with a known schema. ODBC drivers connect SQL Server to most legacy databases including Access, FoxPro, dBase, Paradox, and older versions of Oracle, DB2, and PostgreSQL. The middleware reads or writes through the ODBC layer, which means the legacy application does not need to be modified at all. PCG's audit phase verifies the legacy schema and identifies which tables and columns are safe to integrate against versus which require additional handling.

Scheduled file exports are the second approach when direct database access is not available or carries operational risk. The legacy system already produces reports, exports, or batch files for human review. Middleware watches the folder where those files land, picks them up automatically, parses the format, and loads the data into the destination system. This pattern has the advantage of leaving the legacy system completely untouched. The middleware sees the data only after the legacy system has already finished producing it.

Screen scraping and process automation are the third approach for systems with no programmatic interface at all. PCG uses this approach reluctantly because it is more brittle than direct database or file integration, but it works for legacy applications where the only available interface is the user interface itself. The middleware drives the legacy application through scripted interactions that mimic what a human operator would do, then captures the data the application displays. This approach fits specific situations and is documented carefully because it depends on the legacy interface remaining stable.²

What about transactional integrity when data moves across system boundaries?

Transactional integrity across system boundaries is the integration concern that distinguishes production-grade middleware from script-and-cron approaches. A flow that writes to two systems must succeed in both or roll back from both. Operations that process one record at a time must not lose records when a destination system rejects part of a batch. Updates that fan out a customer record across three downstream systems must reach a consistent state across all three, not a state where two have the update and one does not.

PCG handles transactional integrity through three design patterns depending on the flow's requirements. The first pattern is two-phase commit, which works when all participating systems support distributed transactions through XA-compatible drivers. This approach is reliable but limited to systems that expose the necessary transaction interfaces, which excludes most legacy and many SaaS systems. PCG uses two-phase commit when available and documents the trade-off when it is not. The trade-off documentation matters because the integration team has to know which guarantees the architecture actually provides versus which it approximates.

The second pattern is the saga pattern, which handles distributed transactions through compensating actions when a downstream system fails. Each integration step records what it did. If a later step fails, the middleware executes the compensating actions in reverse order to bring the systems back to a consistent state. Saga is appropriate when two-phase commit is not available and when the business can tolerate eventual consistency rather than strict atomicity.

A third pattern is idempotent message processing, which makes message queue and event-driven integrations resilient to retries. Each message carries an identifier the destination system uses to detect duplicates. If the middleware retries a message after a transient failure, the destination system processes it the first time and ignores subsequent duplicates. The pattern is the foundation of reliable asynchronous integration and PCG implements it as standard practice on message queue integrations.³

How does PCG validate that integrations work correctly before going live?

Validation is a defined phase of every PCG integration engagement, not an assumption made after deployment. PCG's validation approach runs through three layers, each catching a specific category of issue before the integration reaches production traffic.

The first layer is unit testing on the transformation logic. Every format conversion, every business rule applied during the flow, and every edge case identified during the audit gets a unit test that asserts the expected behavior. The unit tests run automatically whenever the integration code changes, which means the team can refactor the integration confidently as the source or destination systems evolve. PCG delivers the unit test suite as part of the integration deliverable so the team owns the validation capability after the engagement closes.

The second layer is end-to-end integration testing against test instances of the connected systems. Where the source and destination systems offer test or staging environments, PCG runs the full integration against those environments before any production cutover. The testing exercises real data flows with realistic volumes, including failure scenarios where one of the connected systems returns errors. Issues that surface in integration testing are corrected before production deployment.

The third layer is parallel operation during the cutover period. New integration code runs alongside the existing process (manual or otherwise) for a defined verification window. PCG compares the output of the new integration against the previous process and reconciles any differences. The new integration becomes the operational master only after the team confirms the outputs match. Existing manual processes remain available as a fallback during the verification window.²

What does the data movement engagement actually look like?

PCG's data movement and middleware integration engagement runs in four phases, each producing a deliverable the business owns regardless of whether the engagement continues. The phased structure matches the patterns PCG uses across .NET migration and Access modernization work, with adaptations specific to integration projects.

Phase one is audit and integration inventory. PCG documents every system in the current integration portfolio, every manual data transfer the team performs today, every scheduled script that moves data between systems, and every no-code platform connection the business currently relies on. The deliverable is a written integration inventory the business owns as a planning asset.

Phase two is architecture and pattern selection. PCG designs the target middleware architecture, selects the integration pattern for each flow identified in the audit, and produces a phased implementation plan. The architecture document defines what the middleware layer will look like, what each connected system will see, and how the integration portfolio will evolve as new systems join the business. Pattern selection happens flow by flow, not system by system, because a single source system often participates in multiple integrations with different latency and integrity requirements.

Phase three is implementation and integration testing. PCG builds the middleware layer in custom .NET against the approved architecture, implements each integration flow according to its selected pattern, runs unit tests on transformation logic, and executes end-to-end integration tests against staging environments. Each integration is validated against representative data before it moves to the next phase.

Phase four is parallel cutover and verification. PCG deploys each integration to production in a controlled sequence, with parallel operation against existing processes during the verification window. Previous integration methods (manual exports, scheduled scripts, no-code platforms) remain available as fallback until the team confirms the new integrations match the expected outputs. The integration portfolio becomes the operational master only after verification completes.⁴