Large-scale test automation and regression testing solution for Siemens PLM Software
Context & challenge
Siemens PLM Software is a leading global provider of product lifecycle management (PLM) software and services with 69,500 customers worldwide.
Their top 100 customers have used their technology for an average of 18 years – much before many software companies even existed. The product will continue to grow. It has major releases every six months, with patches applied even more frequently. Sometimes there are customer-specific variations.
Also, based on variations of platforms from customer to customer, the product updates must be applied to a variety of operating systems and browsers. The overall number of test cases for such a complex product runs into tens of thousands
While changes such as bug-fixes and new features are successfully introduced in each release, it is almost impossible to complete regression testing of all test cases in the time available before a release date. This results in quality issues encountered by end users and/or delays in releases. Generally, the testing relies on selective testing by subject matter experts. But with the size and complexity of the product, the number of subject matter experts is never sufficient, and their availability is not guaranteed.
It was completely understood and accepted that automation was the answer for this regression testing problem. Automation had been attempted before with some of the best internal resources who knew the product and the automation process. A pilot programme resulted in hundreds of successful test cases.
The real challenge however was to scale the regression automation to cover tens of thousands of test cases. Siemens chose to discuss their requirements with testing specialists Expleo.
Solution & outcomes
Expleo’s solution for this problem of large-scale test automation is the Test Automation FaQtory®.
Scalability through assembly-line
After analysing the problem Expleo concluded that scalability could be achieved by using the ‘assembly-line’ model from manufacturing. The automation work was divided into three ‘stations’:
- Subject matter experts – who know the product, but need not know automation
- Framework designers – who are automation experts, but also need to know the product to a limited extent
- Automation team – who are automation experts, but need
not know the product
All stations have their interfaces with other stations clearly defined, facilitating the overall efficiency of the assembly-line.
The scalability was achieved by expanding the automation team; a simple matter for a quality assurance partner like Expleo, while maintaining relatively small team-sizes of the subject matter experts and the framework designers.
Siemens PLM are actively involved, as the subject matter experts are from Siemens PLM and the framework design team is jointly made-up of Siemens PLM and Expleo personnel.
Tool and utilities
Carrying the manufacturing analogy further, the Expleo team developed a number of tools and utilities for the various teams to increase their consistency and productivity, while also reducing person-dependency.
Output-based pricing
Calibration phase
Multi-year engagement using Agile methodology
The engagement is multi-year but uses the agile paradigm of monthly sprints and incremental development and deployment of the automation suite. This allows for the flexibility of choosing test cases for automation every month, and also keeps the automation suite updated with the latest builds of the product being tested.
Client benefits
True partnership
The approach built a true partnership between the customer and Expleo, with each partner focusing on core competency. The required scale of test automation was possible only because of Expleo’s specialisation in testing and commitment to deliver on automation, one of their key strengths.
Transfer of risk to Expleo
Siemens PLM benefited by paying only for the success. The risk in successful deployment of automation was thus transferred fairly to Expleo, as opposed to a headcount-based engagement model, where the risk is predominantly with the customer.
Output-based pricing incentivises efficiency for year-on-year benefit
But the real benefit of output-based pricing is even more subtle. In a headcount-based model, efficiency may not be even desirable, as it may reduce the vendor headcount. In the output-based model, the vendor finds way to improve delivered output on an on-going basis. Thus, this approach incentivises building efficiency into the process. Siemens PLM can thus expect year-on-year benefits from this approach, either of larger volume of automation or a reduction in cost for the same volume.
QUALITY & TESTING
Software testing & integration
Software Quality assurance and testing are as important as software development to improve the end-customer experience. At Expleo, our teams work together in our best-in-class technological centres all over the world to operate a scalable and cost-effective global delivery model.
