Microsoft's commitment to partner success in Data Platform & Analytics practices stems from our own success in moving our financial systems to Azure. That, and the fact that big data, data warehousing, and advanced analytics are generating sizeable and profitable projects for our partners.
The massive migration of our IT infrastructure is now about 90% complete, with many lessons learned during our journey. We know that getting your customers on board with moving such mission-critical and tightly guarded applications to the cloud can be an uphill struggle.
Finance is historically risk-averse territory, and finance departments are justifiably cautious when asked to try new things. Just ask Robert Venable, principal architect for Microsoft IT. You can hear his story in a just-released video on rebuilding Microsoft's revenue reporting system using platform as a service (PaaS).
Too Big for On-prem
Called MS Sales, the 20-year-old data warehouse and analytics platform built on SQL Server had grown into a pretty large and complex system. Imagine a system that has evolved from tracking customer purchases of packaged software, which typically happened in three to six-year cycles, to tracking 2.6 million transactions a day, from 1,500 sources–everything from large purchases to micro-transactions on Azure. And the team is architecting to handle about ten times that amount of transactions.
The tipping point came a couple years ago, when the data size and growth of MS Sales was outpacing the growth in compute power. According to Moore's Law, MS Sales had 18 months before the wheels fell off.
Venable had two options. His team could lift and shift MS Sales to the cloud using infrastructure as a service, or build it anew using PaaS and big data solutions. Either way, he was heading down a path with an unknown destination. He chose to keep his options open.
Scale Up or Scale Out?
Rather than try to scale up, Microsoft IT chose to scale out with a distributed modern data warehouse running in Azure. The team used open source Apache Spark for HD Insight, which allowed them to use existing code for either streaming or batch processing, and had a robust enough architecture for scaling out.
The goal went beyond moving MS Sales to Azure. The project also held the opportunity to make the application more agile, with the ability to incorporate new business models without cumbersome engineering efforts. Scalability and speed would increase with a reduction in complexity and latency.
Since MS Sales has to run 21 years of data, historically our business users would wait 24 hours for the data to process to see new data. Today, those same users can see fresh data every 42 minutes. To test how that would scale, they increased the data to ten times the volume, and the processing time increased by only 10 minutes.
Venable expects to add machine learning capabilities so business units can do predictive forecasting rather than look at what has happened.
Other benefits of the new platform include:
- Easy-to-deploy rule changes
- Easy-to-manage rule definitions
- Ability to redefine business processes to fit business needs rather than technical limitations
Once they started seeing the benefits, the business units could start thinking about how their world would change, and what it would look like in the future. It allowed Microsoft to stop thinking about infrastructure and light up new business capabilities, rather than maintaining servers.
For more details on how the reinvented system is making dramatic improvements in the platform's performance and scalability, read the technical case study on IT Showcase. For more guidance on your own data platform migrations, download the Data Platform and Analytics Playbook.
Based on MDC Research's Cloud Practice Development Study, the following Azure services were most popular among partners with data platform & analytics practices.
To build expertise and develop a Data Platform & Analytics practice, we offer the following competencies:
Partners can also take advantage of several self-paced free courses relating to data platform and analytics in Azure through our Massively Open Online Courses (MOOCs).
If you've started a data platform migration project, please share your own benefits, surprises, struggles, and solutions below.