So, you have a new product feature to implement. You have your functional requirements and an initial design. Now is the perfect time to start privacy threat modeling.

The Process, Step by Step

1. Gather available information about the system (diagrams, requirements, design documents - interview team members if needed)
2. Create one or more data flow diagrams. For a simple system, you might create just one; for others, it might be more manageable if you create multiple diagrams at different levels of abstraction, with particular detail for high-risk areas such as authentication and authorization.
3. Use a privacy threat modeling framework to help you identify potential privacy threats. (We'll learn about these next).
4. Consider the privacy harms these threats could lead to, group them (multiple threats may lead to the same harm) and assess the risk posed by each threat. (We'll learn about privacy harms and risk assessment soon).
5. Prioritize the threats and decide what you will do about them. For some, it may be appropriate to accept the risk; for others, to transfer the risk to someone else (e.g. a cloud provider) or mitigate it via technical and/or organizational measures. In some cases, the risk may be so severe that you need to avoid it entirely, for example by deciding not to build this feature.
6. Repeat as you continue through the development process! Threat modeling is an iterative process. The Threat Modeling Manifesto describes it as a "journey of understanding". Create a best effort threat model now, but if you find yourself stuck evaluating a particular threat, move on. As you implement and test, the threats will become more concrete to you and you may also identify new threats that you missed before or that have been introduced by requirement or design changes.

You may find it helpful to imagine: what would I do if I were trying to abuse this system? If I wanted to steal data or target a specific user, where would I start? Where looks like a weak point?

Data Flow Diagrams

A data flow diagram (DFD) depicts how data travels through your system. It should show all sources and destinations for data, and illustrate what happens to it between. How is it processed, and which system components (entities, for example a microservice) do this? Is it stored in a cache, database, or warehouse? Where are the trust boundaries? (For example, you might draw a boundary between public resources and resources that require you to be on the corporate internal network to access them.) For privacy, we are focused on the flow of personal data rather than all data processed by the system.

This data flow diagram from OWASP depicts the entities, data flows, and trust boundaries (red) involved in a banking application. Useful additions for privacy threat modeling would be to list the personal data types involved in each flow, the country each entity is located/hosted in, and the retention period for the data store.

It is possible to make DFDs incredibly complicated, especially if you use dedicated threat modeling software to create them, which might ask you to fill in a dozen different fields for each entity. Our advice would be to keep them as simple as possible unless you have a hard requirement to apply some automated analysis. A basic diagram that the whole team understands is much better than a diagram that is perfectly conformant to some complicated specification but is unreadably complex. Start with just one diagram, and if things are getting too messy in one part of the diagram, abstract away that part of the system in this top-level diagram and make a second diagram focusing on the details of that part. A range of tools exist for creating and collaborating on threat modeling diagrams:

• Collaborative whiteboard tools such as Miro (see their data flow diagram template)
• Drawing tools such as Draw.io or Excalidraw
• Dedicated threat modeling tools such as the Microsoft Threat Modeling Tool or OWASP Threat Dragon, which supports one privacy threat modeling categorization (LINDDUN).
• And of course, if you're threat modeling together in person, you can simply use a whiteboard or pen and paper. This does make it harder to maintain the threat model over time, however. It's a living document: your system won't stay static over time, and neither will your threats.

Privacy vs. Security Threat Modeling

This image from the NIST Privacy Framework illustrates how security and privacy risks are distinct, but do partially overlap.

Privacy and security threat modeling are complementary to one another. It's often recommended to conduct them together, but it's important to understand that they're not the same. They focus on different threats and require different domain knowledge.

• In security, we primarily consider threats to organizations and systems. When individuals are considered, it's typically in the context of ensuring that malicious users can't compromise the system, and examining threats posed by employees (such as a malicious insider, or someone who falls for a phishing email).
• In contrast, the privacy threats introduced by a system are primarily threats to its users and their social networks, rather than threats to the system itself. Your privacy-violating system that hoovers up personal data and sends it to data brokers may be perfectly secure: no-one unauthorized can steal that data, but you're still selling it on for nefarious use. However, security is an essential prerequisite for privacy: if you're not in control of information flows because an attacker could easily steal data, then clearly none of that data is private.
• In security, the classic 'triad' of objectives is CIA: we want to maintain confidentiality, integrity, and availability. In privacy, these are important too, but rather as prerequisites that enable privacy objectives. The triad of privacy engineering objectives proposed by NIST consists of predictability, intervenability, and manageability, as shown below.
• Because they focus on different threats, they use different threat modeling frameworks and categorizations. We'll learn about privacy threat modeling frameworks later in this section, while a common threat categorization for security is STRIDE.

NIST's triad of privacy engineering objectives, as a privacy complement to the CIA triad of security objectives.

Where they are similar is that they both require a similar mindset, benefit from beginning as early as possible, and share a starting point of gathering documentation and creating a data flow diagram. How to best conduct security and privacy threat modeling depends on your organizational setup, but one suggestion would be to start with a joint workshop with the development team, a security engineer/architect, and a privacy engineer/architect. After this initial interactive session, the dev team can update the threat model as needed and the security and privacy contacts can periodically review it.

Further Resources

• Threat Modeling Process - Larry Conklin, Victoria Drake, and Sven Strittmatter, OWASP. If you're new to threat modeling, this article describes the traditional security threat modeling process.
• Threat Modeling Security Fundamentals - this free training course from Microsoft walks you through the process of creating a data flow diagram and conducting security threat modeling for a system.
• Experiences Threat Modeling at Microsoft - Adam Shostack (2008). This discusses some of the challenges of getting developers started in threat modeling, such as the fact that many people struggle to "think like an attacker" - as Adam says, this is about as unhelpful as saying "just pretend to be a professional chef" to someone cooking! Consider what would help you do this, or if you find thinking like an attacker easy, think about how you could help others via examples, categorizations, and stories.
• Podcast: Privacy Threat Modeling with DoorDash's Nandita Rao Narla - Skyflow. Key takeaways include that threat modeling is intended to be an educational exercise (you don't need to be an expert to get started!) and that you need to strike a balance so that privacy isn't seen as the "office of no" - data minimization is important, but isn't always the best mitigation strategy.
• NISTIR 8062: An Introduction to Privacy Engineering and Risk Management in Federal Systems - Sean Brooks, Michael Garcia, Naomi Lefkovitz, Suzanne Lightman, and Ellen Nadeau, Information Technology Laboratory (2017). Introduces the triad of privacy engineering objectives.
• Towards a Triad for Data Privacy - Quentin Covert, Mary Francis, Dustin Steinhagen, and Kevin Streff, Proceedings of the 53rd Hawaii International Conference on System Sciences (2020). Discusses the NIST triad of privacy engineering objectives and attempts to improve upon it.