APM (Application Performance Management) has grown to be a multi billion dollar industry with a new vendor showing up every day. With numerous players out there, promising various things, it can be a tedious task to decide on which product to procure.
I’ve seen several APM products at work in my 15 years of experience doing Application Support. I have seen an APM product pin point a production issue within minutes, and on the other end of the spectrum, I’ve seen few APM products actually create production issues (not a situation you want to be in, especially if you are the one who signed off on purchasing it).
I’ve put together a list of 7 critical things you should look for while selecting an APM product.
Here we go.
1. Overhead
Most of the APM products nowadays are agent based, meaning a piece of software (called agent) will run inside your application collecting metrics and sending them to a Server that will process the metrics and deliver them via a client interface (typically, a web browser). CA Wily Introscope, DynaTrace and AppDynamics are three popular examples.
Be absolutely sure that the agent will NOT add large overhead to your application. A CPU utilization overhead of 2 to 3 percent per processor is acceptable but not beyond that. Note that you can easily mis-configure an agent to report tons of data (for example, by an incorrect JMX filter) thereby causing a big CPU utilization spike.
2. Versatile Interface to view the Metrics
How light and versatile the client interface is ? Is it fast and easy to start up ? Or is it something that need to be instrumented with an another APM agent to diagnose what’s wrong with it ? Some APM solutions offer both a thick client ( a desktop application ) and a browser based client (Web Client). AppDynamics actually offers a flash based client (which runs on a web browser). The last thing you want in an APM solution is a hanging APM Dashboard.
3. Complexity of instrumentation
How do you add a new Application to the APM? Is it as easy as turning on a flag by editing a property file that can be done on the fly ? Or is it as hard as a)create a file system for agent binary b)create a file system for agent logs c)come up with a unique name for agent d)edit not just one but several property files e) Configure Application Server specific settings, for example, add a ‘startup class’ in WebLogic f)Log on to APM Server to perform few more ‘new agent’ configuration items….. common, this is insane. Have your engineers analyze the complexity of instrumenting a new application. Think long term – what if you need to instrument hundreds of JVMs or CLRs in a short time period.
4. Performance of the APM engine/Server
APM for the most part has become a number crunching business. The ability of the APM engine/Server to process millions of metrics in a short time (like few seconds) is of paramount importance. Review the capacity requirement of the APM Engine/Server. Does it ask for infrastructure as if you are going to build a search engine to compete with google ? Or is it reasonable ?
5. End to End Monitoring
It is not enough to just monitoring Java or .NET (Application Server). You need to monitor the Database, ERP backends, Mainframe, Security Servers, Web servers, Firewalls, Load balances etc. While I have to tell you that true end to end real time monitoring is extremely difficult to achieve, some vendors like correlsense and optier claim to have done that. I haven’t personally used them but you must perform a rigorous POC to determine if the product can indeed show you the end to end performance data.
6. Configuring Alerts and integrating with existing ticketing system
How difficult is to add an Alert and integrate with an existing ticketing system so that a problem ticket is automatically opened ? How easy is to activate/deactivate alerts ? The last thing you want is having a customer complain about an outage before you find it. So, your alerting system must be top notch and always working.
7. ROI
This is obvious. But you really need to crunch some numbers to compare various APM products. There could be hidden costs that you may not be aware of. Do they offer non-prod license for free when you buy production licenses ? What is the upgrade policy ? What is the support cost ? Is the support 24X7 ? What is the turn around time for support tickets ? Once you have answers to these questions, you will be able to come up with actual cost accurately.
There you have it. While choosing an APM product can be lengthy and confusing, you must have some working APM solution to proactively monitor your infrastructure. Follow the guidelines above and you will be better prepared and be able to make a reasonable decision.