When I get put on a new project I'll make certain adjustments but overall my goal is to get to 'Performance Ready' as quick as possible. This allows me to divide my time on things that really matter. I can focus about 70% of time on exploratory performance testing and syncing on serious defects with developers; while 20% is spent on analysis, and 10% spent on fun things like front end performance or sleeping.
Below are a few notes on how I get to 'performance ready' in a few stages.
Performance testing either excites, or annoys. Its a specialized field which requires above average knowledge in several areas not limited to testing, development, architecture, networking, and social engineering. One can have a pretty solid performance approach in 5 very simples stages.
This is pretty simple. Identify which individuals are your real stakeholders and have the information you need. These individuals may range from obvious roles like: architects, managers, team leads, developers, testers, and of course the product owner. Don't overlook your individual developers or testers-in-the-know who can sometimes yield much more valuable information then leads or managers. Once you have an idea of who really holds the information you need to know about, next keep in mind whom out the group really affects decisions. Another words understand who calls the shots...in some groups for example a development lead may have a larger impact on resolving issues or authorizing [read: demanding] changes to improve performance then a QA manager.
So now we've identified who really matters. Next lets shmooze..
Identify and get wide agreement on suggested test scenarios.
These scenarios should be representative of types of user activities, and groups of users.
When translating assumed customer activities into test scenarios and later into workloads you'll need to consider average assumed use, and extremes cases.
What's the difference between test scenarios and workloads?
A test scenario is a specific scenario in which a feature or part of the application is targeted with a specific functional objective.
A workload is collection of multiple test scripts which represent one or more test scenarios; and are executed to simulate specific application use on mass level.
If your lucky enough to merit logs from customers' servers, or anything that provides analytical details to construct workloads then by all means don't be a donkey...use it. Let's say that your not that lucky...you'll need to sit with your product architect, and development lead and understand the products pain points. Next sit with the Product Owner who deals directly with customers and ask him/her for feed back of which parts of the paths through the application are most common. Ask the Product Owner what features are customers asking for or expecting in this release; what are the most common user cases that customers are using now and will be using in the next release. It's also critical at this point to understand from the product owner how licensing for your application works. All the user scenarios that you have gathered and discussed should then match the licensing structure that your actually selling or believe will be sold. Meaning if your selling licenses for 40 application users, don't test for 20.
Define your data set.
One more time...define that data set and do it well. Without realistic test data, or good customer intel all your hard work will benefit each every customer that resides in imaginary land.
Your work is useless if its not measured against real test data. If you don't know what your customers are doing then ask them. If you can't ask them, then speak with the customer service agents and product managers who are on the front line. If your working on a product where the product owner and manager have no idea what the customers will be doing or what types of environments that they'll have - then you've got bigger issues then performance testing.
Define bug criteria.
If your a performance tester with QA background this step should be self evident. All non-functional testing (in this case performance testing) requires objectives set clearly upfront; otherwise prepare yourself for never ending testing. Break up your performance bug discussion into a few general categories such as but not limited to: Scalability, Optimization, Functional, Degradation.
It's important for scalability and optimizations issues to be based directly off your dataset with assumed growth factored in; which has been agreed to grudgingly by your stakeholders in earlier discussions. Otherwise any good finds will be subject to constant debate. Functional - does it work? - NO? Throw your shoe at the nearest developer of course...just kidding...ok not really. My personal peeve is to begin actual scripting of performance scripts only once I have some small bit of feature confidence meaning the feature is mostly flushed out and might partially work, but the structure itself is not changing by much.
Stage 2: Scripting
At this point most features are now working, the data set has been created, and the test scenarios have been defined in high level. Pull out your favorite test tool and goto work. Through out this stage scripts are being created, and tweaked. Don't miss the opportunity to include juicy functional assertions in your test such that other testers can re-use your scripts for functional testing. I remember once sitting in a meeting hearing "But his scripts focus on performance, we need functional tests not performance tests". I responded by explaining that all performance tests contain functional assertions if they don't then your testing is useless. Functional assertions in performance tests are necessary to validate the actions your executing are causing results that your expecting otherwise whats the point? Silo'ed Testers (who own a specific area or responsibility in a team) sometimes forget that others can really benefit from their scripts. If written in a 'drop in and run' style there's no reason why a simple Jmeter script can't contain numerous functional assertions and be created dynamic enough to run against different environments to benefit others.
Stage 3: Defining Workloads
Didn't we just discuss this? -- RULER SLAP - no...the focus of this stage is to consider your previously discussed test scenarios from a top down approach. Scripts should be grouped into flows. These flows are assigned activity frequency, concurrency, and think times. Designing workloads is sometimes overlooked and rushed. Running misleading workloads which are not accurate produce results that aren't accurate either.
Designing complex workloads is really a larger discussion. For those interested see this piece on UCML or just about anything from Scott Barber http://www.ibm.com/developerworks/rational/library/5219.html
Keep in mind this is where you organize, specify concurrency and type of work that should be accomplished in a set period of time. You might have workloads that represent fail over testing, stress, and different types of loads. I prefer to organize my scripts to set specific http requests that consist of a user activity as a transaction. This means if a user performs a "Log in" then all relevant http requests that are executed during a "Log In" are collected as a 'transaction'. The following is an example of http requests organized as a 'transaction' in the script level.
When I'm organizing a workload I'll keep in mind how many 'transactions' are intended on the following levels:
Script level: How many transactions per single execution
Expected Workload level = transaction number of single instance X total V_users per hour
Then in Stage 4 you'll actually create the workload and test your expected vs intended results.
This is really key...below I'll explain why.
Stage 4: Scripting Workload
The purpose of this stage is to 100% verify that the workload and scripts contained within are executing exactly as intended. Meaning the number of transactions, functional assertions, and test results are all going as planned. It's necessary to run simple scenarios limited by duration, amounts of VUsers or both to verify that the tests are really going as planned. If your running data driven tests where randomization can be removed, meaning your reading values from a csv; make sure to run 1 VU on the script level to check to make sure all your data is being used as intended. Why? Lets say your searching for "I stopped reading the crappy post 10 mins ago!" well if your regular expression or assertion is not expecting to be passed a non-alpha-numeric character eg "!" then you maybe in for unintended behavior.
This step might seem obvious but if skipped you might find yourself doing the walk of shame back to development to explain that the P1 level defect that you were screaming about was caused by your scripts not the product.
Stage 5: Performance Ready
Performance ready consists of executing workloads, analyzing results, and communicating results out...then repeating. I like using this simple name because managers understand that the testing cycle is now in a stage which implies that the necessary performance ground work is in place and testing is ready to be executed.
Executing Workloads
In order to squeeze every drop of value out of your performance scripts its necessary to have a framework in place that allows for flexible execution and result management. Several exist depending on your tooling.
If you want to know if your framework is sufficient enough or whether to move to something else ask yourself the following:
- Can I push a button and go home to my family and come back to work the next day with results?
- Are my results persistent, organized, and in an analyzable (thats definitely not a word) without much background work required before or after runs?
- If my tests fail in execution or processing results, can I still recover results the following day or later? Meaning your work isn't lost but recoverable.
- Is my process visible, understandable, and usable by all team members?
- Is result/issue notification clear enough that anyone on the team can resolve it besides the weird guy with stank breath?
- How happy is everyone within the team with the framework? Are there any steps outside of analysis once in a 'performance ready state' that are performed manually and increase frustration and result turn around time?
Analyzing Results
Outside of scripting most of your time is probably spent here. You'll need different reports for different stakeholders. Your ability to efficiently analyze results will be dependent on the maturity of the framework in which your results are managed. Generating reports on the fly for specific stake holders is important but not as important as generating results that are useful to YOU.
Most likely your average developer, team lead and manager could give 2 hoots about what a specific transaction's median, 90% value,max and min may be for a certain transaction. But you my dear performance tester you do care...it will make your life easier when you have the metrics you need at your finger tips.
The point is...automate the creation of reports that you need to make quick decisions first. Leave automating cute charts for later until you have what you need. It's possible in your analysis of results the the numbers show the workload's aren't achieving the intended level of stress, or could be overshooting it. If that's the case go back to redefining your workload to make it accurate. Involve the necessary folks to think things out so that what your running and what people think your executing match up. The worst possible situation is to find yourself testing things that don't matter, while things that do go overlooked.
Communicating Results
Communicating results in timely manner is critical. Don't get bogged down on generating pretty graphs, save yourself time by creating templates for sending out the information. Additionally it may be helpful to gather all the stakeholders and give a brief demo of how to interpret performance results from the templates that you send out. You may find out that each technical and non technical audience per project is different. There also maybe easier ways to communicate more effectively by understanding what your audience immediately understands vs what they need to have explained. Managers may want high level graphs, developers specific response times, architects may want everything - your ability to present clean, accurate results will build your cred' - if you need further convincing check out Mike Kelly's article on "What's your credit score" http://www.testingreflections.com/node/view/4945 'nuff said eh?
If your looking for a thorough book on performance testing check out this online book Pattern and Practices: Performance Test Guidance for Web Applications This book contains a ton of great information that makes it an easy read, in addition to a nice reference. Many of the stages and ideas discussed in this post are covered in detail, with varying chapters on implementation per methodology.
No comments:
Post a Comment