Behind the Scenes: We Report from one of Creo 1.0’s Sprint Reviews

In a previous article, we described Scrum, the internal process PTC uses to develop Creo. For our research and development teams, Scrum provides a progressive approach to successful project management. But does it matter outside the lab—to the people who use the software after it’s released?

Michael Pfrommer, VP of Creo Product Development, says Creo customers benefit greatly from Scrum because it ensures releases are useful and reliable.

In this continuation of our interview, Pfrommer describes in depth the testing that goes into Creo and how we decide when to release:

GH: I’ve often heard the phrase that PTC ‘builds in quality’ during software development – what do you mean by that?

Pfrommer: At PTC, we encourage our software developers to “Built in Quality” as they design and develop the functionality. We spend time up front making sure the requirements are clear, functional designs are reviewed and approved by the team, and developers pre-test their code changes in their local development environment before submitting back to the main software development stream.  We really try to drive to build in quality right at the design and coding.

GH: Previously, you mentioned that during each development period or “sprint,” the team creates a potentially shippable product. That is, everything added during the sprint works correctly before you proceed with the next set of new features or fixes. Could you describe the testing for each sprint?

Pfrommer: You’re right, it’s critical each sprint is functional and tested. So after just a few weeks of development, we test the code. Individual checks are rung against any new or updated fuctionality.

GH: Randomly? Methodically?

Pfrommer: Strategically. Five high-level steps make up most testing cycles:

  1. Reaching agreement. First, we agree on what will be tested, i.e., the test plan. This typically comes from a mix of product requirements, definition, and acceptance criteria.
  2. Documenting our tests. In this step we document those tests, test cases, pass conditions, and, if possible, build an automated test.
  3. Running the tests.
  4. Reporting, tracking, and manage the test results.
  5. Identifying failed tests and prioritizing them.

GH: Do you automate a lot of the testing?

Pfrommer: Yes. All quality engineers continually develop automated tests and build ever -increasing test suites that test all aspects of the sprint. For Creo, we execute over a million automated tests before a release goes out.

GH: How do you digest all those test results?

Pfrommer: For each test, the result is either a Pass (the test and result ran, and the results were correct), Fail (the test ran, but the results were incorrect) or OK to Fail (the test ran, and the results were analyzed, and the actual test needs updating). This summed up across all the tests performed for each sprint gives you an objective quality view for each sprint.

GH: And after you’ve completed a number of sprints, you have a potential release candidate?

Pfrommer: Yes. The Product Owner affirms that the product requirements are met with the completion of a sprint and nominates that version of the code as a release candidate.

GH: Why isn’t every sprint considered a potential release candidate?

Pfrommer: Well, it’s a mix of factors. The Product Owner must decide if the software is sufficiently complete to meet bring value to our customers. Release too early and the product won’t meet needs; release too late and customers may have found another solution, so there may no longer be any opportunity for PTC.

We also ask how has the marketplace evolved? How has the business evolved? Perhaps we’ve acquired a technology or product that can help us meet the customer’s needs better, or a partner has developed a solution in that space.

Finally, we consider that every time we nominate a potential release candidate it enters a broader phased testing, where we involve more people. In the broader testing, we include more internal people, software partners, and customers. This all takes effort, and we really only want to do that when we can be sure we have an impactful release.

GH: Can you discuss the steps in that broader phased testing?

Pfrommer: We look at the whole product in its entirety, for a release candidate, we’ll:

  • Consider more aspects beyond functional quality, for example, performance, reliability, usability, scalability, serviceability, and compatibility.
  • Run tests on a wide range of machines and platforms, using the release candidate, outside of the developer’s build environment.
  • Involve a lot more manual testing, for example, by releasing a beta version of the software that can be shared with testers around the world.
  • Test compatibility with our other products.
  • Test all the different languages and supported language configurations.
  • Release a version of the release candidate to our software and hardware partners so that they can certify their offerings with our latest release.
  • Track all the above, reporting, tracking, managing, and addressing critical issues.

GH: So the broader testing is as important as the sprint tests.

Pfrommer: Right; both matter. I see three key benefits to the Scrum approach.

First product stability throughout development. We want to make sure our customers are excited and look forward to each release, and within each sprint we see rapid progress in product evolution. It’s a transparent and step-wise approach – so every few weeks we can immediately see results and demo them to our customers.

The second is the recognition that during a project we can change our minds about what is wanted and needed. With the new technologies in Creo, Scrum provides the platform to deal with the unpredicted challenges that cannot be easily addressed in a traditional predictive or planned manner.

Finally, with the completion of each sprint, we have a potential candidate for release.

GH:  How much emphasis does PTC put on the testing phase of product development overall?

Pfrommer: I can tell you that there are over 1 million automated tests, and that 20% of our total product development capacity is dedicated to testing. I can tell you that for Creo 1.0, we’ll invest over  200,000 hours on manual testing, and that without counting the thousands of hours that will be conducted by our customers, software and hardware partners, and other external people.

GH: With all this automated testing, will there ever be day when there’s no manual testing?

Pfrommer: No. I can’t imagine that.  We can minimize issues in a number of ways. For example, going to the apps concept will reduce complexity, involving fewer lines of code. But we’ll always test, and we’ll always need a warm body to use the software to find the quirkiest and, sometimes most obvious, flaws.

Watch out for our next Behind the Scenes articles as we review the current status of Creo 1.0.

This entry was posted in Behind the scenes and tagged , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

2 Comments

  1. results
    Posted Jan 19, 2011 at 12:21 pm | Permalink

    OK, well..um…what specifically was learned from the completion of this sprint? I really didn’t gain much insight to creo development from this piece…

    how is the GUI code working out? How is the explicit modeling code and parametric reconciliation going? etc.?

  2. Geoff Hedges
    Posted Jan 19, 2011 at 1:33 pm | Permalink

    Hi results, thanks for your comment. We’ll be dedicating an article during February that reviews the latest sprint for Creo 1.0.

10 Trackbacks

  1. […] team met for a scheduled sprint review of Creo 1.0. You may recall from previous articles here (and here) that a sprint is a short development cycle, part of the agile process we use at PTC for Creo. The […]

  2. […] ways we collect feedback* during our development process. For example, we devote many resources to quality testing to ensure Creo does what it’s  supposed to do. But April’s on-site beta test focused more on […]

  3. […] de développement s’est réunie pour une revue de sprint de Creo 1.0. Cet article, et celui-ci, vous rappelleront qu’un sprint est un cycle de développement court, qui fait partie de la […]

  4. […] sprint review di Creo 1.0. Come descritto in precedenti articoli (consultabili facendo clic qui e qui), uno sprint è un breve ciclo di sviluppo nell’ambito del processo agile utilizzato da PTC […]

  5. […] pendant la phase de développement. Par exemple, nous allouons de nombreuses ressources au contrôle qualité pour nous assurer que Creo exécute ce qu’il est censé faire. Ceci dit, les tests bêta […]

  6. […] Review für Creo 1.0 zusammen. Vielleicht erinnern Sie sich von früheren Artikeln her (hier und hier), dass es sich bei einem Sprint um einen kurzen Entwicklungszyklus handelt, der Teil des agilen […]

  7. […] Entwicklungsprozesses Feedback* einholen. Beispielsweise investieren wir viele Ressourcen in die Qualitätsprüfung, um sicherzustellen, dass Creo die erwarteten Funktionen erfüllt. Beim Beta-Test im April stand […]

  8. […] di raccogliere feedback* durante il processo di sviluppo. Ad esempio, dedichiamo molte risorse al testing di qualità per garantire che Creo fornisca i risultati previsti. Ma i test beta in sede del mese di aprile […]

  9. […] Creo 1.0 の スプリント レビューを行いました。以前のこちら (およびこちら) の記事でご紹介しましたが、スプリントとは、PTC が Creo […]

  10. […] が仕様どおりに機能することを確認するために、多数のリソースを品質テストにあてています。しかし、4 月のオンサイト ベータ […]

Post a Comment

Your email is never published nor shared. Required fields are marked *

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <strike> <strong>

  • Archives

  • Connect with PTC Creo