Software Engineer II - AI Infrastructure (Scheduler) - CoreAI

The AI Platform organization builds the end-to-end Azure AI stack, from theinfrastructure layer to thePaaSanduser experience offeringsforAI application builders,researchers, and major partnergroupsacross Microsoft.The platformis core to Azure's innovation,differentiationandoperationalefficiency, as well as the AI-related capabilities of all of Microsoft's flagship products, fromM365andTeamsto GitHub Copilotand Bing Copilot. We are the team buildingtheAzure OpenAIservice,AIFoundry,Azure MLStudio, Cognitive Services, and the global Azure infrastructure formanaging the GPU and NPU capacityrunning the largest AI workloads on the planet.

One of the major, mature offerings of AI Platform is Azure ML Services. It provides data scientists and developers a rich experience for defining, training, fine-tuning, deploying, monitoring, and consuming machine learning models. We provide the infrastructure and workload management capabilities powering Azure ML Services, and we engage directly with some of the major internal research and applied ML groups using these services, includingMicrosoft Research and the Bing WebXT team.

As part of AI Platform, the AI Infra team is looking for a Software Engineer II - AI Infrastructure (Scheduler) - CoreAI, with initial focus on the Scheduler subsystem. The scheduler is the "brains" of the AI Infra control plane. It governs access to the GPU and NPU capacity of the platform according to a complex system of workload preference rules, placement constraints, optimization objectives, and dynamically interacting policies aimed to maximize hardware utilization and fulfill greatly varying needs of users and the AI Platform partner services in terms of workload types, prioritization, and capacity targeting flexibility. The scheduler's set of capabilities is broad and ambitions. It manages quota, capacity reservations, SLA tiers, preemption, auto-scaling, and a wide range of configurable policies. Global scheduling is a distinctive major feature that overcomes the regional segmentation of the Azure compute fleet by treating the GPU capacity as a single global virtual pool, which greatly increases capacity availability and utilization for major classes of ML workload. We have achieved this capability without allowing a major global single point of failure, based on regional instances of the scheduler service interacting via peer-to-peer protocols for sharing capacity inventory and coordinating handoff of jobs for scheduling. Our system manages significant amount of GPU capacity even outside Azure datacenters, through a unified model and operational process and highly generalized, flexible workload scheduling capabilities.

To be able to manage the inherent complexity of the Scheduler subsystem and enable it to meet the stringent expectations of high service reliability, availability, and throughput, we emphasize rigorous engineering, utmost precision and quality, and ownership-from feature design to livesite. Quality mindset, attention to detail, development process rigor, and data-driven design and problem-solving skills are key for success in our mission-critical control plane space.

• Work on the design and development of the core AI Infrastructuredistributedand in-clusterservices that support large scale AI training and inferencing.

• Develop, test, andmaintaincontrol plane services written in C#, hosted onService FabricorKubernetes(AKS)clusters.

• Enhance systems and applications to ensure high stability, efficiencyandmaintainability, low latency, tightcloud security.

• Provide operational support and DRI (on-call) responsibilities for theservice.

• Develop and foster a deep understanding of the machine learning concepts, use cases, and relevant services usedby our customers.

• Collaborate closely withserviceengineers, product managers, and internal applied research and data science teams within Microsoftto build bettersolutions together.

• Investigateuseoftools andcloud servicesand prototype solutions for problems in ourcontrol planespace.

• Embody ourcultureandvalues.

Required Qualifications

• Bachelor's Degree in Computer Science or related technical field AND 2+ years technical engineering experience with coding in languages including, but not limited to, C++, C#, Java, Scala, Rust, Go, TypeScript

  • OR equivalent experience.

Other Requirements

• Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
  • Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Preferred Qualifications

• OOPproficiencyand practical familiarity with common code design patterns

• 2+years of experience withservice developmentin a distributed environment,ina dev-ops role,including concurrencymanagementand stateful resource management

• Master'sdegree in Computer Scienceorarelated technical field

• Hands-on experience with public cloud services at the IaaS level

• Advanced knowledge of C# and.Net

• Proficiencywithuse of complex data structures and algorithms, preferably in the setting ofa resource allocator/scheduler,workflow/executionorchestrationengine, database engine,or similar

• Significant experiencewithunit testing andwriting testable code

• Technical communication skills: verbal and written

• First-hand experience with building large-scale, multi-tenantglobal services with highavailability

• Experience withbuilding and operating"stateful"and critical control plane services; handlingchallenges with data size and data partitioning;relateduse of a NoSQL cloud database

• Experience with mapping complex object models to relational and non-relational datastores

• Dev-ops experiencewithmicroservices architecturein acomplexinfrastructure andoperational environment

• Service reliability and fundamentals engineering; instrumentation for KPIs or performance analysis;demonstratedservice and code quality mindset

• Performance engineering: work on scalability,profiling;CPU,memoryand I/Ouseoptimization techniques

• Applied knowledge of Kubernetes: service model,workload packaging and deployment, programmatic extensibility(CRDs, operators);or equivalentknowledge of Service Fabric

• Server-sideWindowsprogramming andperformance engineering

• Data analytics skills,in particular withKusto

• Experience working in a geo-distributed team

#AIPLATFORM

#AICORE