Blog
Cal's Blog
 |
Cal Evans Director of PCE |
| Cal Evans is the Director of the PHP Center for Expertise at Ibuildings. He has been involved in IT development and management for the past 26 years and for the past 9 has developed primarily in PHP and MySQL. Cal is the author of a book and many articles on PHP development and can regularly be seen speaking at PHP conferences around the world. Cal blogs regularly at http://blog.calevans.com |
Monday, August 31. 2009The Problem With Benchmarks
Sometime around the early 90's I started noticing that major companies like Microsoft and Oracle started adding provisions to the EULA's stating that the licenses forbade users from publishing benchmarks about the product. Evil Big Brother implications aside, I now am beginning to at least see why they think it is a good idea.
The fact of the matter is benchmarking today's web based computer software is almost impossible. In past years, the software was easier to isolate and the number of variables could be controlled so that a useful metric could be produced for buyers to be able to compare two similar products. Today however, when software is increasingly being deployed on the web, there are too many moving parts that have to be taken into consideration. In any modern web application web servers, database servers, third party APIs and the network all have to be factored in when benchmarking an application. Any one of these not functioning correctly could cause your benchmarks to report false data. Benchmarks won't show you the truthGiven that you can't isolate the software you are testing from its supporting pieces, at best you are showing that, given a specific configuration, including network and network traffic, this is how the complete system performs. This is exacerbated by the problem that any given benchmark only shows how the entire system performs the task being benchmarked under the conditions of the test. If your benchmark is attempting to show real-world performance of a database server, showing it performing a tight loop of reads or writes will not provide any useful data. The opposite side of that coin is, showing a complete web application, running under heavy load, will not produce any reasonable results either because the database may not be strained, even if the web server or network is. Benchmarks can show you useful informationBenchmarks are useful however in a certain case. After each major milestone in an application's development and maintenance lifecycles has been released, a comprehensive set of benchmarks should be run on the application and the results charted against previous versions. Assuming an isolated system that does not change, this should show you areas where your application has either sped up or slowed down since the previous version. Used in this manner, benchmarks are a useful tool for system administrators, project managers and the QA team to keep a handle on the performance aspect of the applications. (Remember, "it feels slow" is not an objective analysis of a system's performance) ConclusionBenchmarks themselves are not bad and I don't mean to suggest they are. Actually, it's the faith that people put in benchmarks when making major decisions like what database server to purchase or what programming language to deploy that is bad. Any given benchmark shows you only how that particular system (all moving parts combined together) handles the specific task being tested. If your application is a simple script that runs a for/next loop one million times and then exits, then a benchmark that shows that will be helpful, otherwise, its not. Use benchmarks for what you know they are good for, testing the performance of your system as it grows. When it comes to making technology decision, there is no substitute for rolling up your sleeves and spending some time testing that technology in real-world situations. Share         Trackbacks
WinCache, Apache and a pretty graph!
I really wanted to focus this post on WinCache, rather than OpCode Cachers in general – but it turned into a general post with a star contender. From the Windows Cache extension for PHP website as Microsoft.com: Windows Cache Extension for PHP is...
Weblog: Technologist For Hire
Tracked: Feb 09, 20:24 Comments
Display comments as
(Linear | Threaded)
Hear, hear! and nicely written.
This has been a point I've been trying to make for a while as well. Why you can load stress your application, it really only helps to provide you information on where the bottlenecks are within your own application -- information useful for diagnosing where to spend time and effort, but not necessarily information for basing a decision regarding the selection of an application or its technology stack. For those, many other factors come into play -- the expertise of your team, the hardware you already have, etc.
I am about to start developing a small application for a global client to coincide with a release of one of their products and allows users to register the product and their details (which communicates with the companies CRM system using SOAP) and has some other db intensive operations, in a multi-step form/wizard approach.
The client has mentioned that the number of visitors could be upwards of 100k per day. Do you have any suggestions for how to go about bench marking or load testing the application to see whether it will be able to cope?
Clients never seem to be able to give the right information, so you need to gage this for yourself. Relying on 3rd party software should never be a direct dependency, as such I'd queue requests to the SOAP service (If feasible with the client).
As for database intensive operations, simply profile and optimize. Index appropriately, or break the queries apart. You would be surprised what constitutes a well optimized query. Load Testing: 1. Ensure your hosting provider can handle the traffic. (different data centers) 2. Tools (too many to list), iostat, vmstat, top, ps, iotop, tcpdump/maakit, etc... 3. Five tips for load test planning 4. [url=http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress testing.html] Performance vs load vs stress testing [/url]
Interesting post Cal. As I was reading I anticipated you to honing in on a specific example that's been on my mind lately. A post I've got in the hopper picks on Rasmus' "Hello World" benchmark. It continues to be a popular one amongst frameworks but I believe it's lost a lot of meaning since he first introduced it. Frameworks like Doo that have come out since the benchmark was introduced go to great lengths to absolutely kill on "Hello World" achieving an order of magnitude more "performance" over more established frameworks like ZF. A message I wrote to a mailing list explores the problems with this in more depth. The point and the problem with the Hello World benchmark is that it traces a single path that is supposed to represent a baseline.
When Rasmus introduced "Hello World" he set out to compare a group of world-class cyclists by having them compete in a 100 meter dash. It was interesting, then, because none of the cyclists trained for the 100 meter dash. The result was a generic measure of framework athleticism. Since the hello world 100 meter dash has grown in popularity new sprinters have joined the mix and, not surprisingly, they dominate the cyclists. Unfortunately there is no "Tour de France" PHP benchmark to give the 100 meter dash results any perspective (and, arguably as you point out in your post, the more systems behind a benchmark the less meaningful a benchmark becomes). At the end of the day the performance of real world applications tend look more varied like a Tour with a large number of routes, trips to the database, complex application logic, non-trivial views, etc, than they do short, repetitive sprints to break the 100m "Hello World" finish line. The PHP community would benefit from the perspective a Tour de France of benchmarks, or at least some measurement that pits the Usain Bolts, like Doo, on racing bikes against the likes of the Lance Armstrongs, like CodeIgniter. (Full disclosure: I am a contributor to the Recess PHP Framework)
Cal is perfectly correct, many factors account for the performance of any application. There are many forms of benchmarking and scenarios that can be tested such as profiling, stress, soak testing etc. Performance is also not the same as capacity, and the total throughput of any application.
Bottlenecks can/will occur due to misconfiguration, poor code, and may only arise due to unexpected spikes, usage or through time/changing trends. As a quick example with disk IO. Filesystem choices such as ext3, xfs, reiser offer differing limits, flexibility and performance. Filesystem configuration options, raid etc, all have varying implications that affect the performance of any application. None of which are application specific. Benchmarks will only highlight a very small part within your application, and will ultimately come down to capacity planning, ensuring that the application will be capable of delivering to your/clients needs. What benchmarks wont tell you are usage and trends, particular sections of the site that burden or negatively affect the overall performance. This can be very difficult to gage because you cannot test aspects of your application in isolation easily. Each component will have a differing affect on the overall infrastructure/application. |
