The NATF Questionnaire: Deciding which questions to use (part I)

Note from Tom: If you’re only looking for today’s pandemic post, please go to my new blog. If you’re looking for my cyber/NERC CIP posts, you’ve come to the right place.

This is my second post on the NATF questionnaire. As I said in the first post, I think it’s a very important document, but it’s certainly not perfect. Its biggest problem is that it has too many questions – by my count, there are about 230, and some of these will require a lot of work to answer (moreover, as discussed below, each of these questions has to be answered 3 times, so there are actually 690 answers required). A supplier is going to have to put in a significant amount of work to answer this questionnaire, and this will be a very significant burden for small suppliers.

But my objection isn’t really the sheer number of questions. My objection is threefold:

1. A large number of these questions don’t address important risks to operations, and specifically to the BES.
2. There are important risks that I and my clients have identified that aren’t in NATF’s list, although we believe they’re important to address with questions. More generally, every NERC entity faces its own set of risks, and they shouldn’t feel they can’t ask any more than what’s in the NATF questionnaire.
3. As I pointed out in this post, asking unnecessary questions increases compliance risk for CIP-013 (which I’ll elaborate on in part III of this post). So don’t ask any question that doesn’t address an important risk to operations.

I’ll discuss each of these points in turn. In this post, I’ll discuss the first point; I’ll discuss the other two in parts II and III respectively, coming soon to a blog near you.

I’m assuming you’re using the NATF questionnaire either for compliance with NERC CIP-013-1, or else more generally for your program to address cyber security of your operational technology supply chain (either in the power industry or in another critical infrastructure industry like gas pipelines or oil refining). As I’ve said before, I don’t think you should ask any supplier a question if it doesn’t address an important risk to the BES. No risk (or not a significant risk), no question.

Just about every cyber security questionnaire I’ve seen addresses primarily IT security risks. While those certainly need to be addressed, they shouldn’t be addressed in the same questionnaire as OT security risks. But how do you identify an IT risk? For every question in the questionnaire, I asked myself “Is there a real possibility that, if a supplier didn’t mitigate this risk, there could be an impact on the BES?” And when I say “real possibility”, I mean “If you score the likelihood of this risk as low, moderate or high, is the likelihood of a BES impact moderate or high?”

The main difference between IT and OT risks is that the former focus on the confidentiality and, to some extent, the integrity legs of the CIA triad, whereas the latter focus on the availability and, to some extent, the integrity legs. This is because the primary goal of IT security is to protect data stored in IT systems, while the primary goal of OT security is to protect the availability of OT systems. This isn’t to say that protecting data is unimportant with OT systems, since a supplier might be holding confidential information on those systems (e.g. how they’re configured or their IP address), but the fact is that OT systems are in place to operate machinery, etc. in the real world, not to store and process data.

Therefore, after first running through the questions in the questionnaire (I consider the questions to start with the “Qualifiers” section and include everything beyond that), I divided them into the following groups:

The first group was questions based on the NATF Criteria v1. I consider all of the Criteria address significant risks to the BES, so I had already incorporated them into my list of questions (and also my list of risks). However, I found that in some cases NATF’s wording of the question was an improvement on the wording of the criterion itself, so in those cases I used NATF’s wording. I counted 57 of these questions in the NATF spreadsheet.[i] One example of these is “Do you implement encryption or technologies to restrict access to and obfuscate data in transit (e.g., cryptography, public key infrastructure (PKI), fingerprints, cipher hash)?” This is question DATA-03, which addresses criterion 42.

The second group was questions in the NATF spreadsheet that I identified as addressing significant risks to the BES (by asking myself the likelihood question above), that aren’t included in the NATF criteria. Some of these were already identified in my lists of risks and questions, although even in those cases I often combined my question with NATF’s (I found a number of cases where I didn’t think NATF’s wording adequately described the risk). I counted 26 of these. An example of one of these is “Have your systems undergone third-party penetration testing?” This is VULN-18.

The third group is “essay questions”, which I discussed in this post. I listed three of these, but there were more that fell into one of the other groups, so this is an undercount. My guess is there were 10-20 in all.

The fourth group is duplicates, of which I found four (i.e. they were almost the same as other questions in the same spreadsheet – perhaps they were drawn up by two different people).

The fifth group is general business questions, such as “Describe how long your organization has conducted business in this product area.” (COMP-04) These are general business questions which don’t address risks at all. Of course, they definitely need to be asked, but I assume that every NERC entity already has a set of standard questions like this that they already ask all suppliers, whether OT or not. Business questions should be in a general business questionnaire, not one that assesses OT risks. I counted 16 of these.

The sixth group is product feature questions, such as “Does the computing system support client customizations from one release to another?” (CHNG-11) Of course, it’s very important to ask questions like this before you buy any product for any purpose; this is because you need to decide whether the product meets your purpose. Again, I can’t imagine that any NERC entity isn’t already asking questions like this anyway; they need to be incorporated in a questionnaire on product features, not this one. I counted 25 of these.

I do want to point out that there are some questions that on the surface appear to address security risks but really don’t; they also fall in the above category – i.e. these are really questions about product features, which happen to be security features. An example of this is “Does your computing system support role-based access control (RBAC) for end-users? (Depending on type of computing system, this may be your users internally, or potentially client users of your product.)” This is IAM-26.

Of course, RBAC is always a good capability to have, from a security point of view. But it certainly isn’t always necessary. For example, if there are only a few people who will ever be allowed to access a particular system, it’s obviously a waste of time to go through the process of creating a special role for them and assigning only that role to this system; it’s much easier just to list those people as the only ones who can access the system. There’s certainly no problem with having this question in your standard feature questionnaire, if it’s not there already. But including this question in your CIP-013/OT risk assessment questionnaire both wastes your and the supplier’s time and increases your compliance risk, for no good reason.

The last category is questions that address legitimate security risks that aren’t likely to have an impact on the BES. This category is by far the most numerous, with 97 questions by my count. Almost all of these are “IT” questions that would be perfectly legitimate in a questionnaire used for IT suppliers, which I don’t think should be included in an OT-focused questionnaire. I’ll give a few examples of these.

First example: Question THRD-01 reads “Describe how you perform security assessments of third-party companies with which you share data (i.e., hosting providers, cloud services, PaaS, IaaS, SaaS, etc.). Provide a summary of your practices and/or controls that assure the third party will be subject to the appropriate standards regarding security, service recoverability, and confidentiality.” Of course, this is a very important question to ask a supplier of say data services. They’ll presumably store some of your data, and they may well do it in the cloud. But I can’t think of a data services provider that would be considered to provide services for BES Cyber Systems, which is of course what brings a service vendor into scope for CIP-013.[ii]

OT providers sometimes will want to store information on your systems or networks that relate to the BES. In general, I think you should push back when they ask to do that (in fact, one of my questions is whether they will need to store data at all, and if so whether they will talk to the NERC entity first and conduct a risk assessment, before doing so). If they convince you that they definitely need to store this data and their plan for mitigating the risks looks adequate, then you should give them permission. But if you have questions about cloud security, you should just forbid them to store the data in the cloud, period.

Second example: Question CSPM-04 reads “Does your organization have a data privacy policy that applies to your computing systems?” Again, this would be a good question for an IT services supplier who is likely to have data that needs to be kept private, in this case especially data regarding your employees’ health, financial information, etc. But unless you’re storing personal health information on systems within your ESP (!), this question has close to zero relevance to the BES.

A third and last example: Question CHNG-06 reads “Do you have a systems management and configuration strategy that encompasses servers, appliances, and mobile devices (company and employee owned)?” Once again, this would be a very important question to ask any supplier of IT or data services – in those cases, you definitely want to make sure that the supplier manages configurations of all their servers, appliances and mobile devices. But given that an OT supplier is unlikely to need to store much data from you – and if they do, you will want to have some sort of agreement with them that addresses how they will protect whatever data they ask to store – IMO this risk, while real, isn’t likely to have significant impact on the BES.

But of course, if you disagree with me on any of these questions and you believe they do address significant BES risks, by all means include them in your questionnaire! The point is that you shouldn’t assume up front that every question in the NATF questionnaire addresses a significant BES risk.

Before I leave this point (and this post), I want to mention that the questions in the NATF questionnaire are actually all “times three” (i.e. there are actually 690 questions, not 230). This is because each question has to be answered three times, once for “Supplier Corporate Systems”; once for “Supplier Product” (i.e. whatever you are buying from them); and once for “Supplier Product Development Systems”. Of these, the most relevant for OT/CIP-013 purposes are the latter two. If you were to tell OT suppliers just to answer for those two areas, they would “only” have to answer about 450 questions. That’s still a lot!

But beyond the sheer number of questions, I think it’s a bad idea to ask a single question for all three areas. This is because the question will often need to be worded differently for each area. The risk will be different in each area, and therefore the question needs to be different as well.

As an example of this, let’s look at CHNG-03, which reads “Do you have a process to assess and apply security patches in your environment within a predetermined timeframe?” Let’s think about how it applies to each of the three areas. For Supplier Corporate Systems, your concern is that the supplier is regularly patching all systems in the company. This question might be fine for that area.

However, for Supplier Product, this question makes no sense. The real question is whether the supplier will provide patches to the NERC entity on a regular basis or else within a certain (short) amount of time (of course, there are a number of other NATF questions - based on the NATF Criteria - that directly address patching for products. As I said, I've already incorporated every one of the NATF Criteria into my list of questions).

For Supplier Product Development Systems – which could pose a substantial risk to the BES if someone attacked them and planted malware or a backdoor in a product– the basic question format would be OK, although it would be important to ask more pointedly what exactly is their timeframe for applying patches. If it’s say six months, you would definitely want to talk to them about this!

Any opinions expressed in this blog post are strictly mine and are not necessarily shared by any of the clients of Tom Alrich LLC. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Are you working on your CIP-013 plan and you would like some help on it? Or would you like me to review what you’ve written so far and let you know what could be improved? Just drop me an email!

---

[i] Although there isn’t always a one-to-one relationship between my questions and the NATF criteria. For example, I decided that most of the criteria that deal with incident response plans could be combined into one question, of the form “Do you do each of the following…?”

[ii] I’ll admit this is kind of a complicated question, and I might be missing something in making such a blanket statement. If anybody knows of a provider of data services that could actually be considered in scope for CIP-013, I’d love to hear about it (no name needed, of course).