Long-running work is where enterprise systems either earn trust or lose it. A payment can wait a second. A provisioning workflow cannot. It has to survive restarts, partial failures, slow dependencies, and the messy reality of operators trying to make sense of what happened while the clock is still running. This is the hard part of distributed applications, not the happy path. The teams that get it right do not treat orchestration as background plumbing. They treat it as a product with explicit behavior, especially when something goes wrong.
Anant Agarwal is a Principal Engineer and Principal Member of Technical Staff at a global cloud enterprise software provider, as well as an IEEE Senior Member. His operating principle is simple and opinionated: a workflow engine should make failure boring, because recovery should be designed, not negotiated in the middle of an incident.
Recovery That Is Deterministic, Not Heroic
If a workflow framework is doing real work, it will meet real outages. Industry surveys consistently show that significant outages can cost well over $100,000, with some exceeding $1 million. That is why retry logic and recovery semantics cannot be left to whatever each service happens to implement. You want one place where “what happens next” is knowable, even when the cluster is not behaving.
Agarwal led the design and development of a Dynamic Workflow and Distributed Task Orchestration Framework for a top cloud microservices company, built to execute long running workflows reliably across Kubernetes. It supported dynamic workflow definitions, retries, suspend and resume, and recovery with effectively exactly-once execution guarantees, and it powered mission-critical SDDC components including NSX, vSAN, and VCF. The platform reached 99.999% availability, orchestrated tens of thousands of concurrent workflows daily, improved throughput by 30%, and reduced operational overhead through automation and deterministic recovery. He still comes back to the same bar: if an operator cannot predict what the system will do after a failure, the system is not done.
“Exactly-once is not a slogan; it is a promise you can test,” he says. “When a node dies mid step, the engine should resume cleanly, and the operator should be able to explain why.”
From Visibility to Fast Diagnosis In Complex Systems
Building on the idea that recovery should be routine, the next failure mode is slower and more expensive: teams lose time because they cannot agree on what they are seeing. Tool sprawl is real. In one large 2024 survey, 89% of groups reported using between two and 10 observability technologies, and 15% said they use even more than 10 across their company. When signals are split across too many places, the first hour of an incident turns into log hunting and screenshot swapping, not diagnosis.
In his current role at a large cloud enterprise software provider, Agarwal pushed for a standardized observability platform built on AWS CloudWatch so teams could stop reinventing dashboards and start sharing the same operational picture. He focused on coverage first, expanding monitoring coverage by 80% so the platform behaved consistently across services. He also worked with cross-functional architects to make resilient, testable patterns easier to adopt in day-to-day builds. The result was faster troubleshooting because engineers were starting investigations from the same baseline, not debating whose dashboard was right.
“Observability only helps when it is consistent enough to argue less. If every team instruments differently, you spend your incident budget on confusion instead of fixes.”
Test The Orchestrator Like It Is A Product, Not A Library
After you can run workflows and observe them, the next question is whether you can ship changes without fear. In modern CI and CD environments, median workflow duration across all branches is 2 minutes and 50 seconds, and median recovery time after a software change is under 60 minutes. Those numbers sound fast until you remember how many times a day teams repeat the loop. When orchestration logic is central, a regression is not local. It ripples.
In a previous role, Agarwal built a Docker based microservices testing system that reduced test cycle time by 40% and improved release reliability, then defined platform-wide functional testing standards that improved release consistency and reduced incident volume. This was not about chasing perfect coverage. It was about building a habit where every change to scheduling, retries, and recovery semantics had a way to prove itself before it reached production. His impact on these core systems was also recognized with a promotion from Senior Engineer to Staff Engineer, a signal that reliability work was being treated as business critical engineering, not maintenance.
“Fast tests are not a luxury, they are how you keep the platform honest,” he says. “If you cannot validate recovery behavior in a repeatable way, you will eventually learn it the expensive way.”
Workflow Definitions That Stay Stable As Requirements Move
Once testing and release discipline are in place, the next bottleneck is usually not compute, it is coordination across systems that do not agree on how work should flow. Manual glue work adds up. A 2024 business systems survey found organizations spending 25 hours a week on manual data entry or reconciling data across apps, and using an average of 10 different digital business solutions to run operations. That is the environment where workflow definitions become a product, because every handoff and exception turns into extra human effort.
Earlier in his career, Agarwal built a BPMN 2.0 automation engine to remove manual workflow dependencies and make execution predictable as processes evolved. He reduced design overhead by 60% by turning repeated workflow patterns into reusable definitions instead of one-off implementations. In parallel, he led the architecture and development of a configurable metadata-driven platform used company-wide across SAP HCM products, so teams could extend behavior without rewriting core systems. Those choices mattered because the HR modules served millions of global users, and change had to be safe, repeatable, and understandable to the next engineer who inherited it.
“Workflows fail in the seams, not in the happy path. If you make the definition explicit and recoverable, teams can change requirements without breaking operations.”
Looking Ahead: Workflow Engines As A Cost And Trust Boundary
From that point onward, economics start to matter as much as mechanics. Worldwide public cloud end-user spending is expected to total $723.4 billion in 2025, cloud computing sales are expected to rise to $2 trillion by the end of the decade, and data center capex could total $6.7 trillion through 2030. As more of that spend flows into complex, distributed operations, orchestration becomes a boundary for both cost and trust. Efficient scheduling and deterministic recovery are not just engineering preferences; they decide how much capacity you need and how confidently you can change production.
Agarwal’s work sits squarely in that future. He is a co-inventor of the U.S. Patent “Event services modeling framework for computer systems”, and his career has repeatedly focused on turning distributed complexity into predictable execution that teams can operate and evolve.
“The goal is not to build the most clever engine,” he says. “The goal is to make execution predictable enough that teams can move fast without breaking production.”
