A weekend race to keep CMS email flowing
In an intense race against the clock over a three-day Labor Day weekend, Leidos teams pulled off a massively ambitious project: Teasing apart the shared email services of tens of thousands of government users, and then re-establishing those accounts in two newly separated environments, with all connections and ancillary services intact.
The unusual challenge arose from the fact that the Centers for Medicare & Medicaid Services (CMS) didn't have its own email services environment, but rather shared email services with its parent agency, the Department of Health and Human Services (HHS). To tailor the environment to its specific needs without impacting HHS's email services, CMS made the decision to transfer its email services into a separate environment.
A three-day sprint
All of HHS and its agencies operate email under Microsoft 365, and setting up CMS as a new, separate Microsoft email “tenant"—or as a separate service—wasn't in itself an especially difficult prospect. The bigger challenge was disentangling the 17,000 CMS users' accounts from those of HHS's 12,000 users, all of which were lumped together under a single shared tenant. The team also had to ensure that when the CMS accounts were transitioned to the new, separate tenant that both HHS and CMS users all retained the various types of connections and shared services between the two agencies, such as file-sharing.
It was a lot like brain surgery. Even Microsoft told us they didn't think it could be done that quickly.
Barry Kohn
Senior Project Manager
Normally, a project that complex might require weeks to complete and would inevitably entail some post-transition confusion and temporary loss of access for users. But neither of those were options in this case. Because tens of millions of Americans depend on HHS and CMS for services related to their health and well-being, the transition had to be done over a long holiday weekend to minimize the interruption. And CMS users returning to work had to be quickly back up and running with full access restored.
To pull off the groundbreaking switchover, Leidos brought together teams of Leidos IT specialists, along with CMS IT staff, Microsoft, and several outside contractors. The technical work began well ahead of the weekend, with the preparation of two new data centers and the Microsoft cloud resources needed to provide a home for the 17,000 mailboxes and the 450 terabytes of data that would go with them.
The Leidos teams worked throughout the holiday weekend to make it all happen on time. “If our key people got three hours sleep, they felt lucky," says program manager Joe Weis.
Email services, mail flow, data objects, directories, phones, and Microsoft tools like Endpoint Manager, Teams and SharePoint—we had to make sure everything aligned. There were lots of times we had to write new code on the fly to get things working right.
Mark Hall
Program Manager
The work was far from over once the cutover to the new tenant was made. Now all the Microsoft Outlook user profiles had to be re-established in the new environment, along with several other components. Normally users would have to do much of that work themselves, but the Leidos teams had set up tools to automate those tasks over the weekend.
A user-focused approach
The team credits much of the project's success to its pre-and post-transition user preparation and support.
Leidos collaborated with CMS to inform all users that email would be down over the three-day weekend and helped set up alternative communication methods for emergencies. “It was unprecedented for a government agency the size of CMS to go down for three days," says senior engineering manager Corwayne Pyle.
When the CMS offices opened on Tuesday, users' phones and computers started providing them with start-up instructions, thanks to a special messaging application that Leidos had arranged for all users to pre-install. Most users were able to follow the instructions to get up and running in minutes.
Users who encountered difficulties were able to connect to a support center via video meeting, where they were immediately routed to a video chat room with other users who had encountered a similar problem. A Leidos support specialist addressed the specific issue at hand in the virtual meeting. “We had 85 percent of users up within six hours," says Kohn.
By mid-day Wednesday the User Support Service Desk queue was down to zero, while real-time video support remained available with little or no wait. And at an all-hands CMS meeting on Thursday, the news was all good. “We got nothing but positive feedback from users and senior leadership," says Weis. “Getting those kinds of compliments two days after a major cutover is unheard of."
