class: center, middle, inverse # Big picture software testing ## Unit testing, Lean Startup, and everything ## in-between \* \* \* ## Itamar Turner-Trauring --- # How should you test your software? ## 1. Wrong answers from my early years ## 2. A better model for testing ## 3. Common testing mistakes --- # How should you test your software? --- # How should you test your software? ## Early years: ad-hoc manual testing ??? * it compiles, ship it * shipped broken code * correct code eventually broke --- # How should you test your software? ## Early years: ad-hoc manual testing ## Later: 100% automated test coverage ??? * Not always sufficient * rewrite code many times -> rewrite tests * 100% coverage -> contrived --- # How should you test your software? ## Early years: ad-hoc manual testing ## Later: 100% automated test coverage ## These are inflexible answers! --- # How should *you* test your software? ## It depends! ## On your situation ## On your goals ??? medical device vs. video game 1000s users vs. none --- # A model to help you decide
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
??? will explain later for now, dimensions --- # Example: a new startup ??? * no product * no users --- # Example: a new startup ## 1. Idea ### A new way to do tax preparation --- # Example: a new startup ## 1. Idea ## 2. Test the idea ### Lean Startup's "minimal viable product" ### Build MVP in order to learn if idea is viable ### Show MVP to potential users: are there any customers? ??? * not my favorite methodology * minimal product to see if customers care --- # Example: a new startup ## 1. Idea ## 2. Test the idea ## 3. Learn about users ### Success: you can actually get users! ### Most users have specific tax requirements --- # Testing for Understanding Users
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Example: a new startup ## 1. Idea ## 2. Test the idea ## 3. Learn about users ## 4. Build real product based on new knowledge ### Create specification, what you intend to build ### Implement the specification --- # Product specification ## High-level ### User can file taxes so long as their situation matches tax form 1040-EZ ## Low-level ### `calc_tax(income, deduction) → correct amount` --- # New testing goal: correctness ## Make sure what we built matches specification: ### Users *can* file taxes ### Taxes *are* calculated correctly --- # Automated tests can't prove correctness... --- # Automated tests can't prove correctness... # ... but humans *understand*, humans provide *meaning* ## Tax accountant can check calculation ## Human QA can try to file taxes ## Another programmer can review your code --- # Testing for Correct Functionality
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Humans ## - Adaptable ## - Inconsistent --- .pull-right[ # Computation ## - Brittle ## - Consistent ] # Humans ## - Adaptable ## - Inconsistent --- .pull-right[ # Computation ## - Brittle ## - Consistent ] # Humans ## - Adaptable ## - Inconsistent ## - Slow ## - Expensive --- .pull-right[ # Computation ## - Brittle ## - Consistent ## - Fast ## - Cheap ] # Humans ## - Adaptable ## - Inconsistent ## - Slow ## - Expensive --- # Automated tests prevent change ## Tax calculation doesn't change ### → Yes, write unit tests as soon as tax accountant signs off ## Redoing UI once a week ### → No, changing all the time, don't want to prevent change ## UI is stable ### → Yes, write (a few) automated end-to-end tests --- # Preventing change has costs ## Cost of writing and rewriting tests ## Different parts of codebase stabilize at different times ### Implementation details: always changing ### Prototype: always changing ### New UI: may take a while to stabilize ## 100% automated test coverage isn't useful metric --- # Testing to Prevent Change
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
??? Unit tests More broadly, any sort of automated tests: integration, regression, etc.. --- # Example: a new startup ## 1. Learn about users (from MVP) ## 2. Built first iteration of product ## 3. Learn some more by running in production --- # What happens when you run your software in production? --- # What happens when you run your software in production? # It crashes! --- # What happens when you run your software in production? # It crashes! # Software quickly becomes too complex to comprehend --- # Learn how your software *actually* runs by running it in a realistic environment ## Load testing ## Collect tracebacks & errors from production logs ??? * Pound on your software with lots of requests * This should be automated --- # Understanding Runtime Behavior
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Example: a new startup ## 1. Learn about users (from MVP) ## 2. Built first iteration of product ## 3. Learn some more by running in production ## 4. Learn from users using the new product --- # Back to Understanding Users ## Usability testing ## A/B testing ## Learn about new user needs & problems --- # Software development: a cyclical process
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- class: center, middle # Common mistakes and pitfalls --- # Mistake: correct but changeable
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
??? me as young engineer --- # Mistake: no learning
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Mistake: unchanging but wrong
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Mistake: preventing change, changing code
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
--- # Test based on *your* goal and situation
Learn what to build
Human testing
Understanding Users
←
Understanding Runtime Behavior
Automated testing
↓
↑
Correct Functionality
→
Preventing Change
Build based on learning
??? this model is not the answer! it just gives you some options Takeaway: no right answer, it all depends probably need more than one way of testing at a time definitely across lifecycle of product --- # Slides and cheatsheet ## https://codewithoutrules.com/pycon2017/ # Got testing problems? Ask me anything! ## [itamar@codewithoutrules.com](mailto:itamar@codewithoutrules.com) --- # Q: How do you add automated tests if you don't start with them? ## Different goals and situation require different actions ## Need team and management buy-in and commitment to that idea ## See list of mistakes: lack of automated tests can be a mistake too --- # Q: How do you add tests to inherited code base? ## Book: "Working Effectively with Legacy Code", by Michael Feathers ## Figure out in which layers you want stability ## Add some high coverage end-to-end tests to ensure overall stability --- # Q: Real-world example? ## 1. Manual testing while prototyping ## 2. End-to-end automated tests once I nailed down UX ## 3. Manually test every new feature, then write end-to-end test ## 4. Automated unit tests for some library code, will expand coverage as codebase stabilizes