- •Contents
- •Preface
- •Acknowledgments
- •About the author
- •About the cover illustration
- •Higher product quality
- •Less rework
- •Better work alignment
- •Remember
- •Deriving scope from goals
- •Specifying collaboratively
- •Illustrating using examples
- •Validating frequently
- •Evolving a documentation system
- •A practical example
- •Business goal
- •An example of a good business goal
- •Scope
- •User stories for a basic loyalty system
- •Key Examples
- •Key examples: Free delivery
- •Free delivery
- •Examples
- •Living documentation
- •Remember
- •Tests can be good documentation
- •Remember
- •How to begin changing the process
- •Focus on improving quality
- •Start with functional test automation
- •When: Testers own test automation
- •Use test-driven development as a stepping stone
- •When: Developers have a good understanding of TDD
- •How to begin changing the team culture
- •Avoid “agile” terminology
- •When: Working in an environment that’s resistant to change
- •Ensure you have management support
- •Don’t make test automation the end goal
- •Don’t focus on a tool
- •Keep one person on legacy scripts during migration
- •When: Introducing functional automation to legacy systems
- •Track who is running—and not running—automated checks
- •When: Developers are reluctant to participate
- •Global talent management team at ultimate software
- •Sky Network services
- •Dealing with sign-off and traceability
- •Get sign-off on exported living documentation
- •When: Signing off iteration by iteration
- •When: Signing off longer milestones
- •Get sign-off on “slimmed down use cases”
- •When: Regulatory sign-off requires details
- •Introduce use case realizations
- •When: All details are required for sign-off
- •Warning signs
- •Watch out for tests that change frequently
- •Watch out for boomerangs
- •Watch out for organizational misalignment
- •Watch out for just-in-case code
- •Watch out for shotgun surgery
- •Remember
- •Building the right scope
- •Understand the “why” and “who”
- •Understand where the value is coming from
- •Understand what outputs the business users expect
- •Have developers provide the “I want” part of user stories
- •When: Business users trust the development team
- •Collaborating on scope without high-level control
- •Ask how something would be useful
- •Ask for an alternative solution
- •Make sure teams deliver complete features
- •When: Large multisite projects
- •Further information
- •Remember
- •Why do we need to collaborate on specifications?
- •The most popular collaborative models
- •Try big, all-team workshops
- •Try smaller workshops (“Three Amigos”)
- •Pair-writing
- •When: Mature products
- •Have developers frequently review tests before an iteration
- •When: Analysts writing tests
- •Try informal conversations
- •When: Business stakeholders are readily available
- •Preparing for collaboration
- •Hold introductory meetings
- •When: Project has many stakeholders
- •Involve stakeholders
- •Undertake detailed preparation and review up front
- •When: Remote Stakeholders
- •Prepare only initial examples
- •Don’t hinder discussion by overpreparing
- •Choosing a collaboration model
- •Remember
- •Illustrating using examples: an example
- •Examples should be precise
- •Don’t have yes/no answers in your examples
- •Avoid using abstract classes of equivalence
- •Ask for an alternative way to check the functionality
- •When: Complex/legacy infrastructures
- •Examples should be realistic
- •Avoid making up your own data
- •When: Data-driven projects
- •Get basic examples directly from customers
- •When: Working with enterprise customers
- •Examples should be easy to understand
- •Avoid the temptation to explore every combinatorial possibility
- •Look for implied concepts
- •Illustrating nonfunctional requirements
- •Get precise performance requirements
- •When: Performance is a key feature
- •Try the QUPER model
- •When: Sliding scale requirements
- •Use a checklist for discussions
- •When: Cross-cutting concerns
- •Build a reference example
- •When: Requirements are impossible to quantify
- •Remember
- •Free delivery
- •Examples should be precise and testable
- •When: Working on a legacy system
- •Don’t get trapped in user interface details
- •When: Web projects
- •Use a descriptive title and explain the goal using a short paragraph
- •Show and keep quiet
- •Don’t overspecify examples
- •Start with basic examples; then expand through exploring
- •When: Describing rules with many parameter combinations
- •In order to: Make the test easier to understand
- •When: Dealing with complex dependencies/referential integrity
- •Apply defaults in the automation layer
- •Don’t always rely on defaults
- •When: Working with objects with many attributes
- •Remember
- •Is automation required at all?
- •Starting with automation
- •When: Working on a legacy system
- •Plan for automation upfront
- •Don’t postpone or delegate automation
- •Avoid automating existing manual test scripts
- •Gain trust with user interface tests
- •Don’t treat automation code as second-grade code
- •Describe validation processes in the automation layer
- •Don’t replicate business logic in the test automation layer
- •Automate along system boundaries
- •When: Complex integrations
- •Don’t check business logic through the user interface
- •Automate below the skin of the application
- •Automating user interfaces
- •Specify user interface functionality at a higher level of abstraction
- •When: User interface contains complex logic
- •Avoid recorded UI tests
- •Set up context in a database
- •Test data management
- •Avoid using prepopulated data
- •When: Specifying logic that’s not data driven
- •Try using prepopulated reference data
- •When: Data-driven systems
- •Pull prototypes from the database
- •When: Legacy data-driven systems
- •Remember
- •Reducing unreliability
- •When: Working on a system with bad automated test support
- •Identify unstable tests using CI test history
- •Set up a dedicated continuous validation environment
- •Employ fully automated deployment
- •Create simpler test doubles for external systems
- •When: Working with external reference data sources
- •Selectively isolate external systems
- •When: External systems participate in work
- •Try multistage validation
- •When: Large/multisite groups
- •Execute tests in transactions
- •Run quick checks for reference data
- •When: Data-driven systems
- •Wait for events, not for elapsed time
- •Make asynchronous processing optional
- •Getting feedback faster
- •Introduce business time
- •When: Working with temporal constraints
- •Break long test packs into smaller modules
- •Avoid using in-memory databases for testing
- •When: Data-driven systems
- •Separate quick and slow tests
- •When: A small number of tests take most of the time to execute
- •Keep overnight packs stable
- •When: Slow tests run only overnight
- •Create a current iteration pack
- •Parallelize test runs
- •When: You can get more than one test Environment
- •Try disabling less risky tests
- •When: Test feedback is very slow
- •Managing failing tests
- •Create a known regression failures pack
- •Automatically check which tests are turned off
- •When: Failing tests are disabled, not moved to a separate pack
- •Remember
- •Living documentation should be easy to understand
- •Look for higher-level concepts
- •Avoid using technical automation concepts in tests
- •When: Stakeholders aren’t technical
- •Living documentation should be consistent
- •When: Web projects
- •Document your building blocks
- •Living documentation should be organized for easy access
- •Organize current work by stories
- •Reorganize stories by functional areas
- •Organize along UI navigation routes
- •When: Documenting user interfaces
- •Organize along business processes
- •When: End-to-end use case traceability required
- •Listen to your living documentation
- •Remember
- •Starting to change the process
- •Optimizing the process
- •The current process
- •The result
- •Key lessons
- •Changing the process
- •The current process
- •Key lessons
- •Changing the process
- •Optimizing the process
- •Living documentation as competitive advantage
- •Key lessons
- •Changing the process
- •Improving collaboration
- •The result
- •Key lessons
- •Changing the process
- •Living documentation
- •Current process
- •Key lessons
- •Changing the process
- •Current process
- •Key lessons
- •Collaboration requires preparation
- •There are many different ways to collaborate
- •Looking at the end goal as business process documentation is a useful model
- •Long-term value comes from living documentation
- •Index
246 Speciication by Example
To quote Wes Williams, after collaborating on speciications, “the trust was amazing.” Many companies I worked with use a software development model that’s based on a
lack of trust. Business users tell analysts what they need but don’t trust them to specify it properly and require sign-off on speciications. Analysts tell developers what they need but don’t trust them to deliver, so testers need to ind some way to check independently that developers are honest. Because developers don’t trust testers—they don’t cut code— whenever testers report a problem, it’s marked as impossible to reproduce, or it appears with a note like, “It works on my machine.” Testers are trained not to trust anyone, almost like master spies.
A model based on mistrust creates adversarial situations and requires a lot of bureaucracy to run. Supposedly, requirements have to go through sign-off because users want to ensure what the analysts will do is right—in truth, sign-off is required so analysts can’t be blamed for functional gaps later on. Because everyone needs to know what’s going on, speciications go through change management; really, this ensures that nobody can be blamed for not telling others about a change. It’s said that code is frozen for testing to provide testers with a more stable environment. This also guarantees that developers can’t be blamed for cheating while the system was being tested. On the face of it, all these systems are in place to provide better quality. In reality, they’re only alibi generators.
All these alibi generators are pure waste! By building up trust among business users, analysts, developers, and testers, we can remove the alibi generators and the bureaucracy that comes with them. Collaborating on speciications is a great way to start building up this trust.
Collaboration requires preparation
Although I stipulated that a good way to implement the process in iterations is to hold a pre-planning meeting, I didn’t have anything more to say about preparing for workshops in Bridging the Communication Gap. I introduced the pre-planning phase because we spent too much time at the start of each workshop trying to identify important attributes for a set of examples; the real discussion started once we had something to work with. Now I see that the pre-planning meeting is a part of a much wider practice.
After talking to teams who formalized a preparation phase in different ways, I have learned that the collaboration on examples is a two-step process. In the irst step, someone prepares the basic examples. In the second step, these examples are discussed with the team and extended. The goal of the preparation phase is to ensure that basic questions are answered and that there’s a suggested format for examples when the team starts to discuss them. All these things can be done by a single person or two people, making the larger workshop much more effective.
Chapter 18 Concluding thoughts |
247 |
For teams who worked on projects where the requirements were vague and required a lot of upfront analysis, the preparation phase started two weeks before the collaborative workshop. This allowed analysts to talk to business users, collect examples from them, and start reining the examples. Teams that had more stable requirements started working on examples a few days before, collecting the obvious open questions and addressing them. All these approaches help to run a bigger workshop more eficiently.
There are many different ways to collaborate
I suggested big, all-team workshops as the best way to collaborate on speciications in Bridging the Communication Gap. Again, after talking to teams in different contexts, I know that the reality is much more complex.
Many teams found that, at the start, big workshops were useful as a means to transfer the domain knowledge and align the expectations of developers, testers, and business analysts and stakeholders. But the majority of teams stopped doing big workshops after a while because they discovered that they’re hard to coordinate and cost too much in terms of people’s time.
Once the system is in place, trust improves, and developers and testers learn more about the domain, and smaller workshops or ad hoc conversations seem to be enough to produce good speciications. Many teams approached this from a “whoever has an interest in the story” perspective, involving only the people who would actively work on a task. When the others need to change it, they would learn about what the software does from the living documentation system.
Looking at the end goal as business process documentation is a useful model
If we think of business process documentation as the end goal of Speciication by Example, many of the common automation and maintenance problems disappear. For example, the law in creating overly complex scripts that mimic the way the software is built becomes obvious; scripts always end up being hard to maintain and the communication value of such scripts is marginal.
As a community, we noticed this a few years ago, and many practitioners advised teams not to write acceptance tests as worklows. Although this is good advice for a majority of cases, that doesn’t help when the domain is about worklows, as in processing payments. David Peterson wrote Concordion as a response to all the misuse of worklows in FIT and got a bit closer to the point by advising people to write speciications instead of scripts. Again, it’s a useful rule of thumb but hard to explain to people who deal with websites. The problem is the misalignment
248 Speciication by Example
of models in acceptance tests or speciication and the models in business;1 one small change in the business domain has a shotgun effect on tests, which makes them hard to maintain.
If we focus on documenting business processes, the model in the speciications will be aligned with the business model and changes will be symmetric. A small change in the business domain model will result in a small change in speciications and tests. We can document business processes well before we start writing software, and they’ll stay the same when we change technologies. Speciications that talk about business processes are worth much more over the long term. Business users can participate in documenting business processes and provide much better feedback than they would on acceptance tests that pertain to software.
This also tells us what to automate and how to automate it. It’s easy to spot the laws in changing speciications to include invented testing concepts or it it into user interface interactions. If the speciications document business processes, the automation layer exercises those business processes on software. This is where the technical worklows, scripts, and simulated user interactions need to go. Automation itself isn’t a goal: It’s a tool to exercise the business processes.
In order to create reliable documentation, we have to validate it frequently. Automation offers one inexpensive way to do so, but it isn’t necessarily the only way. Some things, such as usability, can never be properly automated; but we can still try to validate parts of speciications frequently. This addresses the problem of specifying things that are hard to automate, an issue that many teams avoid.
Long-term value comes from living documentation
Almost everyone I spoke with experienced the short-term beneits of faster deliveries and better quality. But teams who “cleaned up their tests” also got fantastic long-term beneits from them. As a consultant, I’ve helped many teams implement these practices, but because I don’t generally stay with anyone for long, I completely miss the long-term effects. Luckily, some of the earliest adopters of these practices have now been using them for six or seven years, and they have seen great beneits in the long term as well.
Iowa Student Loan was able to change a business model quickly because they had reliable documentation. The team at ePlan Services was able to survive the absence of a key team member. The team working on the Sierra project uses “tests” as supporting documentation when they get support requests. At that point, I think it is wrong to call what they used “tests,” because they don’t use them for testing software: They’re documentation that was built to be reliable and relevant.
Most of these teams adopted living documentation by trial and error, when they
1 See http://dannorth.net/2011/01/31/whose-domain-is-it-anyway
Chapter 18 Concluding thoughts |
249 |
were looking for easier ways to maintain tests. They restructured tests to make them more stable, aligning the models in tests and in the business. They restructured the folders containing tests to make it easier to ind all the things that are relevant for a particular change, evolving a documentation system that’s structured in a way that’s similar to how business users think about system features.
At this point I feel relatively conident in making the bold assumption that new teams can get these beneits quicker if they intentionally create a living documentation system rather than arrive there after years of trial and error.
With that in mind, I invite you and your team to try this yourselves. After you’ve tried it, please share your experiences with me. You can contact me by sending an email to gojko@gojko.com.
Appendix A
Resources
Books
Gojko Adzic, Bridging the Communication Gap: Speciication by Example and Agile Acceptance Testing (Neuri, 2009).
Gojko Adzic, Test Driven .NET Development with FitNesse (Neuri, 2008).
David Anderson, Kanban: Successful Evolutionary Change for Your Technology Business (Blue Hole Press, 2010).
Mijo Balic, Ingrid Ottersten, and Peter Corrigan, Effect Managing IT (Copenhagen Business School Press, 2007).
Mike Cohn, Agile Estimating and Planning (Robert C. Martin Series) (Prentice Hall, 2005).
Lisa Crispin and Janet Gregory, Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Professional, 2009).
Kev Darling, F-16 Fighting Falcon (Combat Legend) (The Crowood Press, 2005).
Mark Denne and Jane Cleland-Huang, Software by Numbers: Low-Risk, High-Return Development (Prentice Hall, 2003).
Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software
(Addison-Wesley Professional, 2003).
Steve Freeman and Nat Pryce, Growing Object-Oriented Software, Guided by Tests
(Addison-Wesley Professional, 2009).
Donald C. Gause and Gerald M. Weinberg, Exploring Requirements: Quality Before Design (Dorset House Publishing Company, 1989).
250
Appendix A Resources |
251 |
Capers Jones, Estimating Software Costs: Bringing Realism to Estimating, 2nd ed. (McGraw-Hill Osborne, 2007).
Craig Larman and Bas Vodde, Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum
(Pearson Education, 2010).
Richard Monson-Haefel, 97 Things Every Software Architect Should Know: Collective Wisdom from the Experts (O’Reilly Media, 2009).
Rick Mugridge and Ward Cunningham, Fit for Developing Software: Framework for Integrated Tests (Prentice Hall, 2005).
Mary Poppendieck and Tom Poppendieck, Lean Software Development: An Agile Toolkit (Addison-Wesley Professional, 2003).
James Shore and Shane Warden, The Art of Agile Development (O’Reilly Media, 2007).
Gerald Weinberg, Quality Software Management: Vol. 1, Systems Thinking
(Dorset House Publishing, 1992).
Online resources
Here are the URLs of all the online resources mentioned in the book. You can ind all these links and more on the accompanying website: http://www.speciicationbyexample.com.
Tools
Concordion: http://www.concordion.org.
Cucumber: http://cukes.info.
FitNesse: http://itnesse.org.
GreenPepper: http://www.greenpeppersoftware.com.
JBehave: http://jbehave.org.
Robot Framework: http://www.robotframework.org.
SpecFlow: http://www.speclow.org.
TextTest: http://www.texttest.org.
Twist: http://studios.thoughtworks.com/twist-agile-test-automation/.
252 Speciication by Example
Videos
Gojko Adzic, “Challenging Requirements,”
http://gojko.net/2009/12/10/challenging-requirements/.
Dan North, “How to Sell BDD to the Business,” http://skillsmatter.com/podcast/agile-testing/how-to-sell-bdd-to-the-business.
Hemal Kuntawala, “How we build quality software at USwitch.com,” http://skills matter.com/podcast/agile-testing/how-we-build-quality-software-at-uswitch-com.
Björn Regnell, “Supporting Roadmapping of Quality Requirements,”
http://oredev.org/videos/supporting-roadmapping-of-quality-requirements.
Presentations
Tim Andersen, “Persona Driven Development,” http://www.umsec.umn.edu/ events/Code-Freeze-2010/PDD; http://timandersen.net/presentations/Persona_ Driven_Development.pdf.
Mark Durrand and Damon Morgan, “Creating a Lean business from the inside out: Technical innovation at uSwitch.com to reduce waste,” http://www.slideshare. net/markdurrand/spa2010-uswitch.
Articles
Gojko Adzic, “Agile in a Start-up Games Development Studio,”
http://gojko.net/2010/05/19/agile-in-a-start-up-games-development-studio/.
Gojko Adzic: Are tools necessary for acceptance testing, or are they just evil? http://gojko.net/2010/03/01/are-tools-necessary-for-acceptance-testing-or-are- they-just-evil.
Gojko Adzic, “Examples make it easy to spot inconsistencies,” http://gojko.net/2009/05/12/examples-make-it-easy-to-spot-inconsistencies/.
Gojko Adzic: How to implement UI testing without shooting yourself in the foot, http://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting- yourself-in-the-foot-2/.
Gojko Adzic: Improving testing practices at Google,
http://gojko.net/2009/12/07/improving-testing-practices-at-google/.
Appendix A Resources |
253 |
Gojko Adzic, “QUPER model for better requirements,” http://gojko.net/2009/11/04/quper-model-for-better-requirements/.
Gojko Adzic, “Shock therapy agile adoption at 7Digital,” http://gojko.net/2009/12/08/shock-therapy-agile-adoption-at-7digital/.
Michael Bolton, “Acceptance Tests: Let’s Change the Title, Too,” http://www.developsense.com/blog/2010/08/acceptance-tests-lets- change-the-title-too/.
Michael Bolton, “Testing vs. Checking,” http://www.developsense.com/blog/2009/08/testing-vs-checking/.
Alistair Cockburn, “Sacriice One Person,”
http://alistair.cockburn.us/Sacriice+one+person+strategy.
Craig Larman and Bas Vodde, “Acceptance Test-Driven Development with Robot Framework,” http://code.google.com/p/robotframework/wiki/ATDDWith RobotFrameworkArticle.
Craig Larman and Bas Vodde, “Feature Teams Primer,”
http://www.featureteams.org/feature_team_primer.pdf.
Dan North, “Whose domain is it anyway?” http://dannorth.net/2011/01/31/whose-domain-is-it-anyway/.
Björn Regnell, Richard Berntsson Svensson, and Thomas Olsson, “Supporting Roadmapping of Quality Requirements,” IEEE Software 25, no. 2 (Mar/Apr 2008): 43–47
James Shore, “Alternatives to Acceptance Testing,” http://jamesshore.com/Blog/Alternatives-to-Acceptance-Testing.html.
James Shore, “The Problems with Acceptance Testing,” http://jamesshore.com/Blog/The-Problems-With-Acceptance-Testing.html.
Lance Walton, “Writing Maintainable Acceptance Tests,”http://www.casual
miracles.com/blog/2010/03/04/writing-maintainable-acceptance-tests/.
Comics
Chris Matts, “Real Options at Agile 2009,” http://www.lulu.com/product/ile- download/real-options-at-agile-2009/5949486.
254 Speciication by Example
Training courses
Gojko Adzic: http://neuri.co.uk/training.
Object Mentor: http://objectmentor.com/omTraining/omi_training_index.html. Lisa Crispin and Janet Gregory: http://www.janetgregory.ca/training.htm. Elisabeth Hendrickson: http://www.qualitytree.com/workshops/.
Pyxis Technologies:http://pyxis-tech.com/en/our-offer/training.
TechTalk: http://www.techtalk.at/training.aspx.
Rick Mugridge: http://www.rimuresearch.com/Coaching.html.