An Introduction to SWAT 2.0 August 6, 2008
Posted by Chris in Agile.2 comments
SWAT or Simple Web Automation Toolkit is an open source web automation testing tool developed by software engineers and testers who work at Ultimate Software. Currently SWAT is used in-conjunction with Fitnesse Framework however it is not dependent on FITNESSE being installed, but currently formatting is based of Fitnesse Pipe formatting.
Here are some screen shots of the currently public version 2.0
Version 2.2 SWAT Recorder
Assertion Tool
SQL Assertion Tool
Great Co-Worker Software related Blogs August 5, 2008
Posted by Chris in Agile.add a comment
Check out fellow Scrum Master and Developer’s blog and a lead contributor to open source project SWAT
Also check out Software Architect and Founder of DVDFriend.us technical blog
Exploratory Testing – A Walk Through May 28, 2008
Posted by Chris in Agile.Tags: Exploratory Testing
add a comment
“Exploratory Testing” Defined
An approach to software testing that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project.”– Kaner, Bach. June 2007
No Preconceived Notions
“Testing is exploratory to the degree that you are not bound by preconceived ideas. Testing is scripted to the degree that you are bound to follow what has been designed in advance.” – James Bach
Don’t Do what you always do! Your pre-existing conditioning will dampen your testing efforts. Try to take the beaten path.
Exploratory Testing
-
Is NOT “random testing”
-
Is NOT “unstructured testing”
-
Is NOT a technique, but an approach
-
IS teachable, repeatable, and manageable
-
IS the heart of excellent testing
Missing Requirements?
ET (Exploratory Testing) can be used as a great method for finding missing requirements.
The Difference between a bug and a missing requirement is that a bug is a requirement that is failing, and a missing requirement is one that was never defined.
Helps finds holes in requirements
ET Vs Automation
Automation is great for regression and existing tested functionality
ET is great for discovery and exploring “undocumented” features.
They both can co-exist. A healthy balance of both will increase quality, while a lack of either will decrease it.
Misdirection
Focusing a set test plan all the time may leave you too focused to notice what’s going on outside of the test plan.
-
Are features not in the test plan behaving differently?
-
I break out of a test plan, should I expect a certain result?
-
Inattentional Blindness is a common problem regarding missed bugs.
-
you get too used to your environment, you may miss the obvious.
Tester’s Investigative Tools
-
Observation (5 senses)
-
Interviews (stakeholders, programmers, designers)
-
Screen shots, log files
-
Project documentation
-
Scenario modeling
-
Hunches, inference, feelings
-
Notebook
-
Screen capture, keylogger, video capture, log watch
-
Software programs Test automation
-
Scripting language
-
Monitoring tools
-
sysinternals, perfmon, file monitors etc.
-
Whiteboards, idea diagrams
Transition to ET Strategy
-
Follow script
-
Add variability
-
change input values, or step sequences
-
Stop part way through
-
Stare out the window, take a break, and then interact with the first thing you see
-
Has the time spent been valuable? Should I continue?
-
Lose the script (but not the test purpose)
How do I apply to QA
-
Use a recording tool like SWAT to record your exploratory testing for feedback, regression and future automation (if bugs/ missing requirements found)
-
Assign a story/task for ET to account for time. Report on findings during stand ups to show value.
-
Implement a team tester trade.
-
Have someone not familiar with your page/features to test your product without test cases. Gives fresh eyes and promotes strong ET values.
Can Testers Make a Difference? A thought inspired by Google’s “Testing on the Toilet”. May 13, 2008
Posted by Chris in Testing.1 comment so far
One of my favorite presentations of the whole conference, happened on day 4 regarding Google’s “Testing on the Toilet” keynote. In the testing world this is a pretty popular story, but just in case you haven’t read about it already, here is a link to read.
The idea here is that anyone in an organization can make a difference. It is a common misconception that Testers are an after thought in the development process and only are there to reassure that “Quality” is a term lightly used and rarely influenced. But the real question to ask is, “Why is this so, and How can I(the tester) make a difference?”
Now not every corporate world is the same, so there is no sure fire way to guarantee your message will be heard, and quality will be enforced, practiced, or even noticed. However that does not mean it is not your responsibility to make a change. Here is a few idea’s that a tester can use to inspire ones self to make a difference.
- Start a Quality Control Group
- Rally up the testers, developers, PO’s, Analysts, Marketing or anyone who believes your vision for Quality and wants to move forward to achieving something that everyone can be proud of, and start a forum, whether it be a weekly, month, after work, or on-line meeting where you can brainstorm as a Quality Focused Team to help improve the current process.
- Internal Quality Group
- This focuses on a group within the company, improving the specific processes and figuring out ways to apply this focus on Quality to your individual project/companies projects. This allows for more specific discussions to occur without violating any industry secrets or confidentiality agreements.
- External Quality Group
- This focuses on a group of fellow peers in your local/regionally area that get together and brainstorm on how to improve the industry/field as a whole without necessarily revealing any company kept secrets. This is a great way to see how your peers handle certain scenarios that you may of otherwise been unable to visualize due to the nature of your work environment.
- Internal Quality Group
- Rally up the testers, developers, PO’s, Analysts, Marketing or anyone who believes your vision for Quality and wants to move forward to achieving something that everyone can be proud of, and start a forum, whether it be a weekly, month, after work, or on-line meeting where you can brainstorm as a Quality Focused Team to help improve the current process.
- Let it be Heard!
- Create a Blog
- Create away for people to access and discuss your idea’s. Web 2.0 is a great way of communicating outside of your immediate area. Utilize these tools to put your thoughts into words and let the world contribute to your ideas.
- Contribute to existing Forums, Blogs, and Communities
- You don’t have to be a leader to make a difference. Contributions are what make up any and all changes. Just being apart of the discussion can stimulate new ideas and become a catalyst for change. This is a great opportunity to use existing forums to ask questions, give answers, and learn more about all varying types of topics and how they may relate to you.
- Create a newsletter
You may have idea’s that the rest of peers may not even be aware of. You may be a senior tester yet have no way to distribute your knowledge to the rest of the team/organization/ or even community. By creating a newsletter you can sum up some of your basic ideas and even use it as away to train. Start with a single page, outlining some new feature or test method, give an example on how to implement it, and then follow up with some commonly asked Q&A’s. Send it out to your peers and than ask around for feedback. Others may like to contribute, and in time your readers will become participants and a community will be formed.- “Inspired by Google” – Post in the bathroom, post it in the cafeteria, post it under the build status. Put it where it will be allowed and if its not allowed anywhere, make your case to management that these methods have been used, applied and found very successful.
- Create a Blog
Now go out and make a difference! And just remember don’t stop until you do, not every idea may be right for you, and it will take effort to see it through, but the end result will be a better way, and you will be the reason for it.
Day 4 of StarEast – Automation Testers May 12, 2008
Posted by Chris in Testing.add a comment
- “The Habit of Strategizing”
- This first method is really the way we plan our tests before we actually go ahead and implement them. Do we as testers spend enough time thinking “How can I make these tests more efficient?” Can we combine tests to test multiple levels without having to re-run/re-write an end to end test every time we want to test something. We need to take the time that the proper testing effort is being taken that compliments the product we are testing in the best way. Not all products require the same sort of test tools and testing methods.
- “The Habit of Simplifying”
- This is the idea that we need to do less complex and repeatable tests in order to increase ROI and immediate test feedback.
- Prioritizing the more complex testing on what gives the most coverage first.
- Automating Every Single Possible Scenario usually ends up being time wasted, when stronger more repeatable tests can be created. Also need to consider test maintenance, “Can you maintain the amount of tests you are writing, without spending to much time, updating broken tests?”
- Do the results of the test, clearly point out Pass/Fail?
- “The Habit of Seeking”
- How is the product designed, and how will our test properly test each level?
- Functional (Functional Tests – Object Testing)
- Business Level (GUI – Automation)
- Unit Level (Unit Test)
- “The Habit of Separating”
- Realize the parts of your tests, what are most susceptible to change, and try to separate them from your tests into more manageable pieces. This allows you to update the tests when pieces change such as.
- Data Setup
- Property Changes
- Flow Changes
- Building your tests in pieces allows greater re usability of the test, and also reduces maintenance.
- “The Habit of Storing”
- This is again the focus on re-usability. What is more important, that the Data is re-usable or the tests itself. Will multiple tests share multiple test data? Will this be still an accurate test scenario if so?
- The more your test data and test cases are re-usable the higher ROI on your testing.
- “The Habit of Saving”
- When to test? Ask yourselves these questions before you begin a testing effort,
- Is the product complete enough to begin testing/test planning?
- Do you have enough time to implement testing and gain feedback and changes by the development?
- Is the product stable?
- Can you estimate and plan in advance?
- Do you have requirements?
- “The Habit of Selling”
- Can you give metrics, highlights to why your testing is valuable?
- Send status of your testing results to your team and management for them to understand the Quality of the situation.
- Are your tests shippable. Could you package your tests with production releases as away to run verification on code when it is updated on the client side?
- Be proactive – Don’t sell short Quality, without it your product will most likely fail.
Day 3 of StarEast – Orges are like Web Services (Part 2 – The Sessions) May 9, 2008
Posted by Chris in Testing.add a comment
So for anyone following my blog posts currently about the conference, are asking themselves “How are Orges like Web Services?” To that I must give credit to Brian Bryson from IBM who used it in my first presentation on Testing SOA Applications.
Service Oriented Architecture (SOA) is a computer systems architectural style for creating and using business processes, packaged as services, throughout their life cycle. SOA also defines and provisions the IT infrastructure to allow different applications to exchange data and participate in business processes. – Wikipedia Definition
The above is a small summary of what SOA actually is. To the traditional tester this means a different way of approach compared to traditional interface level testing. As the traditional model will most likely be an Interface wired up to a Web Service, we need as testers to remove the Interface from our testing to truly be able to functionally test the web service. This means,
Gui-less or “Headless” testing. This is the idea of testing beneath the interface layer and testing the web service directly. This means that the functional test must interact directly with the web service, similar to testing of API’s or Object level testing. Here is some of the reasons why you should test beneath the GUI layer.
- Application level validation – The GUI may restrict you from the data you input that does not guarantee that the application/service will be tested to handle bad data. The web services has no input validation it will accept what you send it, but the question is “How will it handle it?”
- Security – Data via an interface is usually malleable and easy to manipulate. This can cause security weakness. Testing beneath the interface allows you to be better aware of how the service handles malicious data.
- Performance – The web can be a bottleneck, testing the application/service beneath the GUI will allow you to accurately get a feel for how well the service itself can handle under heavy load.
So the main way to achieve this testing effort is via automation. This is one of the area’s in software testing where Automation is a must. It is not an easy manual effort and web service components are usually constantly changing, and automation testing will help with the regression testing effort and ensuring that the web service is maintaining its ability to meet its requirements. Below is a few hints on how to test SOA apps.
- Use SOAP Testing tools to gain easy access to Methods to data input.
- Try functional level tools such as Fitnesse to deliver functional level tests integrated with acceptance tests at low cost.
- Try security testing methods like Injection, overflow, malicious data, parameter tampering.
- Boundary Test, The web may limit character intake, but does the method?
- If inter service testing – does the data being sent back always come in a form understandable by the requesting service?
- Load test the service – How does it handle under extreme requests/transmissions? Monitor the CPU/Memory usage on the service machine. Is there memory leaks?
- Data Variation – Try anything. Combinations of Data and values to really understand how the service handles varying sorts of information.
Now that you have some basic idea’s, there will be many variations of tools on the marketplace to help you achieve your testing goals, whether it be open source or license software. The main thing to remember is use your head and don’t follow the traditional path, but the road less taken.
Day 3 of StarEast – Orges are like Web Services (Part 1 – The Keynotes) May 9, 2008
Posted by Chris in Testing.add a comment
The Tutorials are over and now the key notes and speaker sessions have begun. Here is a brief overview of some of the keynotes and presentations that I attended and what I was able to extract from them.
(snippet from his PowerPoint presentation)
• Consider that your …
–Finances, credit and tax information
–Travel documents and records
–Traffic violations and criminal history
–Citizenship, travel history and visa information
–Medical records
–Organization membership information
• …are all stored in and processed by software
The main objective was to express that as software becomes integrated even more into our daily routines and life processes we must realize that Quality cannot be sacrificed and we need to adapt, learn and constantly be ready to change/grow to face the challenges of maintaining Quality in today’s software model.
It is time for Innovation to occur and as an industry we need to move toward improving these skills, tools and processes that we currently use into something bigger, better and more universal. As systems grow the integration between products, companies and people will grow, and Quality is everyone’s responsibility.
Speaking from experience in which I work for an Agile team that implements both XP process and Scrum, I can say this has created highly testable and quality code. Building quality as a team from the ground up with the turn around time of minutes and hours for bugs instead of days and weeks really helps the feeling of productivity and self worth, and makes a very happy team dynamic.
More information about Extreme Programming can be found here: http://www.extremeprogramming.org/
Day 2 of StarEast – Performance Testing May 8, 2008
Posted by Chris in Testing.add a comment
This not being a technical training as much as a training of process and planning for structuring a performance test, I was able to take away some key points, but the most important being; Performance Tests are costly(whether it be time, resources, or tools) and therefore must be planned and set up with significant thought. Here is a list of some of the pre-test planning items that should be taken into consideration.
- Ask not the question “Can we afford the cost of a true performance Test?” but instead “Can we afford the risk of not performing a true performance Test?”
- Definition of True as in the context of the above statement: An exact or very close representation of the production environment in which the product will rest.
- As related to item #1, one must realize that performance tests can be highly influenced by the environment in which it resides. Therefore a test environment should mimic the live environment as best as it can.
- Hardware. Hardware ratio should remain the same throughout the environment as if in production. (ex. If the SQL Server has 8GB of RAM, and RAID HD Setup with 10,000 RPM drives, the test environment should have this set up)
- # of Servers should remain at least in ratio to the production environment. If you have 3 Web Server, 3 SQL Servers, 3 App Servers, your test environment should at least maintain a 1:1:1 RATIO.
- Data. Data should not be a slice of production data it should be a mimic(or production) data. Content may not have to be the same, but the variety and size should be similar/exact.
- Network: Is the application going to be influenced by Network Traffic(Internet, Extranet, IntraNet). If so this traffic should be simulated in the test environment.
- Time. Will there be certain aspects of the application that will be used differently and more frequently during certain parts of the day/year. These should be taken into consideration when determining the expectations and planning of the test boundaries.
- Security. Will there be Firewalls, Throttles, or other administrative tools that will be in production that need to be taken into consideration?
- Bottlenecks – If they exist in the production environment, they must be replicated in the test environment.
- Exceptions to above rules
- Feature Testing/object level performance. Not all performance testing is concerned with End to End/Production level performance. The above rules may not always apply when testing specific load capacities of objects, functions and services. (ex. A application server may not be in control of any outside sources(especially if coming over Internet) and therefore performance may be ran to check how quickly the application server can take data in, process and return the data.)
- When should performance Testing begin?
- In the requirements phase!!! It will always be cheaper to build it in from the beginning then to build it in later. (See Chart below – Cost of Performance Testing). By starting the requirement phase and gathering performance requirements it will be easier to start the Testing process and development process of implementing performance suitable code.
Recommended Link for introduction to performance tools(open source) to start practicing with Performance Testing at no cost(for tools) .
* Training Held by Dale Perry, SQE Training
Day 1 – StarEast – Exploratory Testing May 8, 2008
Posted by Chris in Testing.add a comment
It is my belief(and shared by many others at the conference) that no development/testing practice is not without it’s flaws. One of the main items found in the agile world that seems to be lacking as the direction is moved towards more developer centric automation is the lack of Exploratory testing. Automation is very useful and not to be disregarded as it drives reliability and efficiency of testers during periods of regression and code change.
What the above should relate to than is, through increased efficiency in the test efforts, testers should actually have more time(and should use that time) for exploratory testing. The value with Exploratory testing is that there cannot and will never be a 100% requirements coverage + automative coverage to handle all scenarios of dynamic applications. Even if theoretically possible the time spent would be exponential and would give very little ROI.
Time spent on Exploratory Testing will help drive Quality and Requirements to become more stable and reliable. By spending a few hours whether it be a day/sprint/test cycle to go above and beyond the pre-determined test scenarios to do things that the general user would not do, to try and “break” the application/service can help discover not necessarily new “bugs” but missing requirements that would not normally been thought of. This way through this testing method, you can push forward an enhanced set of requirements that feeds better code delivered by the developers.
As we rely on Automation more and more, I believe we must never forget that as long as we are building a user experience, there will always be a need for User Patterns, and as we all know, users are unpredictable, random, and unquantifiable.
* Training Held by Jonathan Kohl, Kohl Concepts Inc.
