A powerful way to avoid being the next ‘data breach’ headline

Most MFA can still be phished. Here at Datadog we became an early adopter of security keys. Now  almost 5,000 employees, our initial rollout was to 700 employees. To this day, every new employee is issued a security key upon onboarding.

Adopting a new solution amidst IPO

But, let’s take a step back. When we began to craft a plan around adopting security keys as a critical component of our security strategy, it was way back in 2019. Coincidentally, that was the same year the YubiKey Lightning was previewed at CES (Consumer Electronics Show), winning Wired Magazine’s Best Mobile Solution.

Datadog was also experiencing exponential growth…and in a short period of time. We were growing around 60% YoY in staff, and even more in revenue. In addition to all that, we were prepping for an IPO. As I’m sure most IT leaders reading this can relate, we decided to map out and launch the security key solution amidst so many plates juggling in the air.

Mapping out the ROI

We knew our company-wide rollout of strong authentication through security keys would help enhance security -  protecting access to data, applications, and services. These little devices give us an edge over regular passwords or typical two-factor authentication (2FA). Imagine having a lock on your front door, but also adding a high-tech alarm system - that's what using a security key feels like. Even if someone figures out your password, they can't log into services without your security key.

Here's a real-life example: one of our leaders had the unfortunate experience of “SIM Swapping”. In this type of attack, a bad actor convinces your phone company to transfer your phone number to a new SIM card that they control. This scam can give hackers access to a lot of your data including 2FA SMS messages.

Normally, it'd be a scary situation. But because the leader was using a security key as their only second factor, we weren’t concerned with the attacker accessing Datadog systems. Similarly, when one of our GitHub accounts was compromised, we knew our data was still secure because the attackers didn't have the necessary security key. There are a bunch of stories like that, passwords being dropped in online databases, but now it doesn't scare us. All thanks to security keys that keep our company's data locked up tight.

IT as UX works cross-functionally across the organization to deploy policies in an effective way…security is just one example.

The user experience mandate of IT

A natural byproduct of this rollout has been to increase the visibility and impact of IT organization-wide. IT is a very cross-functional role that impacts every team across the company. IT can be seen as the "user experience" (UX) department for employees. What do I mean by that?  IT creates great experiences for the employee and elevates them with technology. We do this while protecting business risks like cost, compliance or security in combination with other departments.  IT as UX works cross-functionally across the organization to deploy policies in an effective way…security is just one example.

It was incredibly important to us that IT helped other teams succeed, but we knew it had to be easy for the end user, otherwise, no one would adopt our solution.  We built a security culture early on with multi-factor authentication, then upleveled with security keys, all with the goal of championing an easy-to-adopt security culture. Security keys would be both an easy and quick win for the Security and IT teams.

Everything is in the cloud, with single sign on you get access to Gmail, and Google Drive and all that stuff - but then you can also single sign on into everything from there, whether it's Workday, or NetSuite, or Azure. It was critical to protect all data. Stop phishing at the source…authentication. With security keys, it’s actually faster to authenticate, no annoying 2FA authenticator apps or text messages. As I mentioned earlier, we created a great user experience company-wide while adopting a security mindset. Here are the steps we took to do so, as a playbook for security key rollout:

The Head of IT Playbook for Security Key Rollout: SMB to Enterprise

Find the correct keys and partner

First, we identified the best keys for us, which we found were dependent on the laptops and devices in our fleet. For us, we have USB-C everywhere so we needed a USB-C key. We also needed a key that works on mobile phones. NFC works across iPhones and Android and USB-C keys can also be physically inserted into some Android phones. There are various keys for different USB ports on laptops, and there are a few NFC keys that also have USB-C on the market.

On the vendor side, there are really only a few vendors, so for us, it came down to the keys and services they provide as well as where the company is located and where the keys are manufactured. Here are a few to keep in mind:

• great cost-effective all-around key that works on desktops/laptops and mobile devices: https://www.yubico.com/product/security-key-series/security-key-c-nfc-by-yubico-black/
• great to keep in your laptop at all times https://www.yubico.com/product/yubikey-5c-nano/

Yubico will even warehouse keys and ship them for you. And they have an API. (The API is used to automate the shipping of keys to people)

Figure out your internal processes

There was a lot to nail down. We knew we had to see how we would distribute the keys to existing employees as well as how to onboard new employees. We had to consider processes for people losing their keys or being locked out because, for example, they forgot to bring their key on a trip with them.

This is all about creating documentation and training - make sure you answer these questions:

• How do you onboard new users on Day 1? What do you train them on? How do you get them the keys if they don’t onboard in person? What are the few main things you want them to remember?
• How do you handle lost keys?
• How do you help people who need to log in but don’t have their key? Regardless of where they are in the world…
• How do you handle apps that don’t support security keys? (there are less these days, but for example, the Workday app does not properly support them)
• What self-service documentation can be created so people can help themselves (if they want)?

Pro tip: Teaching someone to use the NFC key on their mobile device is a bit more tricky than plugging it into their laptop, so we created detailed documentation around that for both iOS and Android

Start slow, but go for high impact

My suggestion is to tackle your highest-risk population first. Which group would cause the most negative impact if their accounts were phished? That’s probably your privileged IT folks (Google Workspace / O365 / Jamf admins etc), the people who administer your cloud service provider(s), or your executive team. Also, administrators of business systems like Salesforce, NetSuite, and Workday are good candidates. Think about worst-case scenarios as far as employee account takeovers, and enroll those people first.

At Datadog, we started with technical employees that had privileged access to systems (think IAM, CSPs, Salesforce, Email) as well as execs. We made sure to include anyone who was featured on our website (primarily C-suite).  From there, we worked out the kinks and slowly rolled out to more people in different job functions. When the kinks were sorted, we went full steam. The sooner employees are protected with keys the better.

Enforce that security keys are the only factor!

If employees still have SMS or TOTP setup, it makes the security keys irrelevant. Our mantra is, “Security keys only!”.  If your end users can still use TOTP or SMS as their second factor, you’re not realizing the value. Make sure security keys are the ONLY second factor they can use. This may require a setting change, a process change, monitoring, or all of the above.

Every time there's some crazy phishing campaign or something going on in the news, we just have that comfort that we're almost certainly not impacted by it.

Educate and sell the ‘why’

Most people still haven't used security keys before and there will be a learning curve and human behavior change required. For end users, we know this is a pretty significant behavioral change. They have to rely on an additional piece of hardware and carry it with them. We’ve baked into the process a step to help them understand that this is really to protect their accounts and data. If they understand why the company is doing this (un-phishable accounts, protect your personal data) it helps them to justify the change in their mind. We’ve taken to saying it over and over in different ways and over different mediums (Slack, Email, Print, in-person meetings, etc).

If you expire passwords regularly now, when you have a security key as a second factor that will no longer be important. So, you can sell the idea of the security key as justifying no longer needing to expire passwords. Also, tapping a security key is easier than entering a code from your phone.

We have a big sticker culture at Datadog so we originally incentivized employees to come pick up their keys by giving them a custom sticker. One funny caveat with security keys is that if you tap them by accident they spit out random text and hit enter (Yubispam). In addition to training and documentation, we made light of this with a custom sticker of the Datadog logo and an OTP code.

Encourage personal use

Security keys are great for personal use too, so we encourage people to use them in their personal accounts and they can keep them even when they’re no longer employed at Datadog. They’re great for protecting banking, social media, and more. The idea is to get them using them regularly so it becomes a more normal part of their life. It’s all about protecting the end user and their data. So, if you can, encourage your employees to use the keys for their personal accounts as well as a small perk. Let them keep the keys when they leave.

Measuring Success

We used our own monitoring platform on Datadog to track (and celebrate) the number of employees in the Security Key program. We considered it a key OKR and ensured company-wide visibility around our goal of achieving 100% adoption success. The tracking is done based on the number of employees in a specific Google Workspace OU (that mandates security keys) versus the total number of employees.

Here’s what a sample snapshot might look like:

It's tough to quantify success because it can be really anecdotal. Essentially, it comes down to peace of mind. Phishing is still the way most companies are getting breached. So we have a lot of peace of mind here that even if a password leaks, these accounts are safe.

Now, if you wanted to get into my Google account, you'd have to know my username and my password, and you'd have to get in physical proximity of me to take this device out of my laptop. It's definitely possible, but that's not the threat model that we're trying to solve for. Every time there's some large-scale phishing campaign or some security incident in the news, we have that comfort that we're almost certainly not impacted by it. We have IT protecting the business against security threats. We have also constructed IT as a great user experience department for every team, working cross-functionally to help them succeed and make an impact. It’s a good feeling.

What's Next:

While security keys as a second factor have added tremendous value to Datadog over the past few years, we are closely watching the development of Passkeys as more and more vendors adopt the technology. With Google recently announcing passwordless login to Google Workspace using “Passkeys”, we are exploring what a passwordless future at Datadog looks like.

Connect with Daniel on LinkedIn to learn more about what he’s up to next.