Find out how to get rid of 80% of silent disasters in actual manufacturing programs with Datadog 

On August 1, 2012, Knight Capital Team deployed a regimen tool replace to its buying and selling gadget. The deployment script failed silently on considered one of 8 servers, reported luck, and no person spotted. Earlier than the workforce understood what was once going down, the gadget had achieved over 4 million trades, gathered $7 billion in unintentional positions, and misplaced $460 million in 45 mins. The corporate ceased to exist in a while after. 

Essentially the most damning element from the SEC investigation was once that 97 computerized alert emails fired earlier than the marketplace even opened. They went unread since the gadget had no observability layer that made the ones indicators not possible to pass over. 

Knight Capital is the excessive model. However the similar trend performs out each week in manufacturing programs throughout industries. Again and again, a deployment succeeds. Previous habits resurfaces quietly. No alert fires that anybody acts on. And by the point somebody notices, the blast radius is already vast. 

That first quantity is the only value specializing in. Maximum manufacturing disasters that stretch customers first are application-layer occasions that creep over days, enterprise good judgment mistakes hiding at the back of 200 responses.  

Those are catchable. Datadog catches them. 

Let’s see how on this weblog.  

What’s Datadog? 

Datadog is a tracking and observability platform. It pulls in the likes of logs and metrics out of your knowledge assets, similar to: 

• Infrastructure  
• Programs  
• Services and products  

The whole lot is united in a single position, so you’ll be able to see what’s if truth be told going down throughout a gadget as an alternative of guessing from scattered dashboards. 

For engineering organizations working allotted, cloud-native stacks, that visibility issues a lot. A request lately would possibly contact a dozen microservices, 3 databases, and a queue or two earlier than it returns a reaction.  

When one thing is going fallacious someplace in that chain, Datadog is helping you hint it again to the supply as an alternative of digging thru disconnected equipment one after the other. 

That’s the baseline. The true query is what occurs when not anything seems to be fallacious in any respect. When a gadget fails quietly, with out an alert, with out an error code, with out any person noticing till a visitor does. That’s what this piece will get into. 

Tracking vs. observability: The hole maximum groups have 

Tracking tells you when a recognized metric crosses a threshold. Observability means that you can ask questions you didn’t suppose to invite whilst you wrote the code. 

Maximum engineering groups have already got one thing. An APM software, some cloud-native logging, a couple of dashboards. However Knight Capital additionally had indicators. The variation isn’t whether or not indicators exist. It’s whether or not your stack surfaces the ones indicators in some way that calls for motion these days they topic. 

Datadog connects all 3 observability pillars into one floor. An error fee anomaly hyperlinks without delay to the logs that give an explanation for it, which hyperlinks to the precise hint appearing which provider name modified after your final deployment and why. No tab switching. No handbook correlation. The solution is already in entrance of you. 

3 silent failure patterns Datadog catches 

Deployment regressions that document luck. Datadog’s deployment monitoring ties each metric and hint to a selected model. The instant a brand new deployment produces a special error trend or latency curve in comparison to the earlier model, Watchdog surfaces it mechanically earlier than a unmarried consumer information a grievance.   

Sluggish queries that creep, no longer spike. A lacking index can take a question from 12ms to 900ms over two weeks as knowledge grows. No threshold alert fires all the way through the creep. Database Tracking surfaces precise execution plans and lock rivalry so that you catch it lengthy earlier than customers really feel it. 

Datadog allotted hint view appearing a complete request lifecycle with flame graph breakdown. Each span is seen together with downstream provider calls and person database queries, with actual timing at every layer. 

97 indicators fired and no person noticed them? Right here’s how one can repair that 

Essentially the most instructive a part of the Knight Capital tale isn’t the $460 million loss. It’s the 97 emails. The indicators existed. The gadget was once telling the workforce one thing was once fallacious earlier than the marketplace opened. The failure was once no longer in detection. It was once in surfacing the ones indicators in some way that demanded motion. 

This issues past alert routing. When infrastructure occasions have an effect on cloud-native monitoring equipment, groups in finding their visibility degrading at precisely the similar second the programs they’re looking at get started failing. Datadog runs as an unbiased SaaS layer with its personal international consumption, decoupled from any unmarried cloud supplier’s provider well being. Your dashboards stay reporting as it should be irrespective of what is occurring on the infrastructure layer. 

What catching a silent failure if truth be told seems to be like 

With out this stack, the similar failure surfaces days later as a enterprise grievance. The blast radius is vast. Remediation takes days. The tale begins to seem so much like August 1, 2012, simply slower and quieter. 

The AI code era issue 

Groups delivery 4x extra code additionally send 4x extra floor house for issues to head quietly fallacious. AI-generated code is assured. It compiles. It passes checks. But it surely makes assumptions about null protection, downstream contracts, and edge circumstances {that a} developer deeply accustomed to the codebase would possibly no longer. 

The Knight Capital deployment script that silently skipped a server was once written by means of one engineer for comfort. No person reviewed it as tool. In 2026, the an identical is an AI-generated migration script or configuration record that appears right kind, passes evaluate, and turns on previous habits in a single atmosphere no person idea to test. 

Datadog’s deployment monitoring manner each model has a paper path for your metrics and strains. Anomalous habits will get correlated to a deployment mechanically. You roll again with proof, no longer intuition. 

How does Datadog compares vs choices 

Why no longer simply construct this on Prometheus and Grafana? Smartly, lots of engineering organizations do. One among our purchasers makes use of Prometheus for metrics and Grafana for dashboards. It really works, and it’s unfastened, in the best way numerous unfastened infrastructure is unfastened.  

However somebody nonetheless can pay, simply in engineering hours as an alternative of a subscription. 

The distance sooner or later presentations up at correlation. Prometheus tells you a metric moved and Grafana means that you can take a look at it. However neither one palms you the hint or the log line that explains why, mechanically, with out a human sewing 3 equipment in combination at 2 a.m.  

Datadog’s pitch is that the sewing is already carried out. New Relic makes a equivalent one. The true query isn’t which platform has extra options. It’s which one will get an engineer from anomaly to root purpose the quickest, at 2 a.m., part wide awake, earlier than the blast radius grows. 

Conclusion 

The top-leverage place to begin is APM with allotted tracing, log correlation, and one Watchdog anomaly observe to your number one API floor. That mixture catches nearly all of silent disasters earlier than they transform incidents. From there, Database Tracking, customized business-logic metrics, and artificial screens fill in the remainder of the image. 

Knight Capital had 97 indicators and no observability. The function is the other: fewer indicators, they all actionable, and a gadget the place your workforce reveals the failure earlier than your customers do. 

Are you additionally working manufacturing programs that can’t have enough money silent disasters? Xavor is helping engineering groups put in force observability that if truth be told works.  

Drop us a line at [email protected] to get in contact with our knowledge engineers. 

Concerning the Writer

Affiliate Director Customized Apps

Umar is a era chief that specialize in cellular software building, with 15+ years of revel in construction scalable virtual answers. He specializes in designing high-performance cellular and internet packages, serving to organizations ship seamless consumer stories thru fashionable applied sciences, cloud platforms, and leading edge building practices.