Application compatibility in the Windows ecosystem

Our application ecosystem is incredibly diverse, encompassing tens of millions of applications (apps) with numerous versions, languages, architectures, services and configuration options. While our ecosystem is complex, our vision is simple. All apps on Windows devices should just work! Customers should update with confidence, never worrying about compatibility across Windows versions. Achieving this goal requires innovative approaches to predict and prevent issues and traditional approaches like testing. In this installment of our quality blog series, Mete Goktepe from our Windows Application Compatibility team describes the various programs and technologies we use to improve app compatibility.

We work very closely with partners to measure, validate and improve application compatibility for every supported version of Windows. Testing is a critical step, but we also know that testing is not enough to fully understand our dynamic ecosystem, so we invest in new methods and product improvements. Once we discover that an OS change has impact to an application or a class of applications, we partner with internal and external developers to implement mitigations in the OS or the application. Because these fixes can take time and our goal is to limit the impact of any issues to our customers, we take specific steps to prevent customers from having a bad experience while fixes are developed, flighted and deployed.

We work across several areas to ensure compatibility:  proactive prevention, issue discovery and investigation, internal and external partner engagements, and compatibility mitigations.

Proactive prevention

Our Windows Application Compatibility team focuses on preventing app issues by partnering with Windows feature development teams to make Windows more compatible and resilient. As teams work on new features for each new release, they bring code changes to the Windows code base. We use static and runtime analysis tools to help us detect changes in APIs to determine if there are any potential compatibility breaks caused by these code changes. We especially monitor changes that may impact 3^rd party, independent software vendor (ISV) apps such as antimalware and antivirus (AV) solutions and then closely collaborate with them to address changes needed before new releases.  Additionally, using machine learning models, we are developing new capabilities to understand how our code changes during active development cycles and build a risk-based recommendation model that prescribes apps we should test. As these models mature, we expect to continuously improve our time to issue detection metrics by tailoring our testing coverage to catch compatibility issues faster.

Issue discovery and investigation

Every time we change our code, there is potential to impact an app. We must balance enabling innovation and raising the security bar in the OS with continuously monitoring and assessing any impact on apps. To maintain compatibility in Windows 10, the team validates thousands of apps internally each week. These apps serve as our issue detection canaries and are chosen based on telemetry (popularity) from the ecosystem or based on risk level and partner feedback. We also employ app selection criteria based on an app’s API usage and its unique way of exercising underlying OS platform capabilities. As the partial list below suggests, we consider many device/form factors and application categories in a complex matrix.

Partial view of categories covered by our internal test matrices

We verify app compatibility continuously; it starts seconds after a new build completes and continues with each flight, quality and feature update. Our multi-layered approach includes the following types of testing:

• Automated application compatibility integration tests – Large and complex projects like Windows require participation and collaboration from thousands of engineers, working on different branches of the source code every day. Before a branch can be integrated into the main Windows development branch, it is subjected to automated tests, referred to internally as “gates.” Our app compatibility gates test the primary functionality of first- and third-party apps. For desktop versions alone, we test hundreds of different apps for every completed build of Windows, across all branches. This type of testing adds up; we run more than 280,000 individual app compatibility tests each day. Results are reported within two hours of build completion, helping ensure that any failures are understood and addressed, and that Windows remains stable.
• Daily automated application compatibility tests – As code changes reach the main Windows development branch, we also run a broader set of automated tests covering around 2,500 apps to detect any new compatibility-related breaks. The results of these daily scheduled test are analyzed on the same day, allowing engineering teams to detect and fix issues that are not caught during prior branch-level integration testing.
• Manual application test passes – We leverage automation heavily today and will continue to invest in additional automation. But we also rely on manual verification to ensure there are no end-user experience regressions. The automated solutions often find functional regressions. But, to validate the experience we often need to experience the application as our end users would. Here are few examples of end user scenarios that we validate manually:
  • Automation can display a menu and make selections from that menu. However, it can’t easily verify that each menu item is visible and usable by the end user.
  • Automation cannot evaluate app behaviors that change when a specific peripheral type(s) is connected/attached to the device.

Data and analytics-based issue detection:  Our teams also make use of telemetry signals to detect and predict potential regressions impacting the app ecosystem with new OS releases or OS flights to Windows Insiders. Through telemetry, we monitor app crash signals closely to determine if there are any increases. Similarly, our automated user feedback monitoring systems sort through thousands of feedback items daily to help us detect potential compatibility issues impacting Windows customers; we make these a priority to resolve.

With this level of ongoing app compatibility testing and monitoring, our goal is to maintain high compatibility pass rates with every release of Windows 10. We execute these tests against both Windows 10 feature updates and then monthly quality updates to help keep our compatibility at the highest levels possible.

Internal and external partner engagements

Finding a compatibility issue through testing or telemetry is just the first step. We also work with partners to resolve compatibility issues and update either the app or the OS. To do this, we partner closely with many internal and external teams to drive resolution for newly discovered compatibility issues during the development cycle. External partners include external ISVs and original equipment manufacturers (OEMs). Below are two examples of external engagements:

Desktop App Assure Program: We are confident in the app compatibility promise we make. That’s why at Ignite in September 2018, we announced the Desktop App Assure Program, which provides app remediation services to our commercial customers as they deploy Windows 10, Office 365 ProPlus and subsequent feature updates. Desktop App Assure service is available to address valid application compatibility issues at no additional cost. The program is designed to:

• assist customers in remediating custom-developed line-of-business apps
• engage 3^rd-party independent software vendors to help remediate their Windows 10 apps and issue support statements for Windows-as-a-Service
• address issues found in Microsoft first-party products (including Office macros and add-ins)

AV vendor engagement: Many Windows 10 devices use 3^rd-party antimalware and antivirus (AV) solutions; these apps sometimes have compatibility issues. Approximately 80 independent software vendors (ISVs), covering +95 percent of the ecosystem, (e.g., Avast, Kaspersky, McAfee, Qihoo, Sophos, Symantec, Tencent, Trend Micro, etc.) participate in our MVI (Microsoft Virus Initiative) program.  We maintain a close relationship with these vendors to provide a “better together” integration experience that protects our shared customers from malicious software. Many AV vendors also participate in our Security Update Validation Program (SUVP) program to validate security updates (discussed in a December 2018 blog). Similarly, we run hundreds of AV products in our compatibility labs to report results on Windows 10 feature and quality updates. We are very committed to continue expanding on collaboration opportunities with AV vendors to maintain highly compatible and secure experiences for our customers.

Compatibility mitigations

Depending on the issue, timeline and the ability of internal or external partners to resolve issues, the Windows application compatibility team evaluates a range of mitigations to prevent customers from experiencing a compatibility issue. These fall into two basic types — shims and blocks. A shim is a method to help an app or feature to function as expected, even if OS functionality may not fully support it or has changed. Blocks are simpler and preclude a user from installing a Windows 10 update that would create a compatibility issue. When we place a block due to a compatibility issue with an app, we will not install the update on devices that have the affected app installed until that issue is resolved. When the issue has been fixed, re-tested and fully deployed, we then remove the block.  We have recently begun providing greater transparency about these blocks on the Windows 10 Update history page. While we have rich capabilities to mitigate issues through blocks or shims, we consider this only for temporary cases and prioritize fixing the issue and updating the OS for a complete resolution.

We are committed to providing fully compatible app experiences on Windows devices. Today we do this through proactive prevention, a multi-level validation approach, engagement with a vibrant and diverse ecosystem and compatibility mitigations. We will continue to innovate in the methods we use, and our goal is nothing short of full compatibility with each update experience.

The post Application compatibility in the Windows ecosystem appeared first on Windows Blog.

Source: Windows Blog
—