Measuring Performance

This page describes some performance testing conducted by CA Technologies.

Performance is a notoriously difficult thing to measure. There are many variables. This page describes only one testing setup. Your mileage will vary.

Performance is usually measured using two broad axes:
  • response time: this is how long it takes for a request to get an answer. This is obviously critical for applications with human users, but may be less critical for machine-to-machine systems
  • throughput: this measures how many concurrent requests the server can handle. If a server can respond very quickly, but can only handle (say) one request at a time, it may become overloaded when a lot of requests come in at the same time, causing delays and even timeouts. If a server takes some time to respond, but can process many requests at the same time, it is less likely to get overloaded, and the performance will be more predictable.
The combination of these two axes will determine the total performance of the server. Ideally we'd like the response time to be as close as possible to zero, and throughput to have the highest possible value.
But we live in the real world, and there are many factors influencing this.

Factors

The performance of your Live API Creator service will be influenced by a number of factors.

Speed of CPUs

Obviously, the faster your CPU(s), the faster Live API Server will run -- assuming the database and the network can keep up.

Number of servers, number of CPUs per server, and number of cores per CPU

Live API Creator is designed to take advantage of as many servers, CPUs and cores as possible. In theory, there is no limit to how many can be used, but in practice at some point some other component will start to be the limiting factor: the database or the network are the most common.

Amount of heap

Live API Creator is (usually) not memory-hungry. How much memory you should allocate to it depends a great deal on the size of your database model(s), how much logic you have defined, and how many concurrent requests you get. For most applications, 1GB of heap per server is sufficient. For smaller systems, 512M is often sufficient.

Latency to the database

Latency to the database measures how long it takes for a query (or insert, update, delete, etc...) to get to the database, and how long for the response to get back, excluding the time it takes to execute the query. Latency is one of the most critical factors in performance. If the latency to your database is more than a millisecond, you may start seeing some performance degradation. It's usually a very good idea to physically place your Live API Server as close as possible to your database server(s).

Query performance (includes all operations, including inserts, deletes, etc...)

This is another critical factor. Live API Creator, by definition, cannot go faster than your database server(s). If a query takes a long time to execute (perhaps because of an inefficient execution plan, or improper indexing), performance will suffer accordingly. This can often be optimized with better queries, better indexing, and other common database optimization techniques.

Complexity of resources

If you define complex resources, with many levels, the retrieval of all that data may take some time. This is not necessarily a bad thing if the alternative would be to make many requests. It's usually better to spend a little time in the server than to make many requests.

Number of rules, and complexity

If you define a lot of rules, and these rules are complex, e.g. they make calls to outside systems, or they make complex calculations or queries, then anything that results in the invocation of that logic will obviously incur its cost.

Complexity of security

If your security settings are complex, e.g. you have defined a lot of permissions for a table, then the execution of the queries may be slowed down due to the increased complexity of the SQL.

Test Environment

When we test performance, we use the following environment. All servers are located in the same room, on a 1GB/s network.

Live API Creator server

This is the server that runs Live API Creator on Tomcat 8.
OS: CentOS6.7 (this changes over time, of course)
8 CPUs

Admin database server

This server runs CentOS 6.7 and MySQL 5.6. It holds the admin database, e.g. the database that stores all of Live API Creator's metadata (resource definitions, rules, etc...)

Database server

This server runs CentOS 6.7 and Postgres 9.4 This is the database that contains the "user" data, in this case a simple schema consisting of customers, orders and line items. The database contains several million rows.

JMeter server

This server runs CentOS 6.7 and JMeter. It runs the testing scripts. This can be quite demanding -- it is not unusual for this machine to run out of steam before the others.




Comments