Roll Your Own, Part I: "Programmable"

Note: This post became the first in a series of posts on this issue.  You can find them all by dropping down the month folders on the right of the screen.

Prologue

I recently realized that I have progressed through all of the stages of grief regarding the problems in CIP-002-5: Denial (not seeing these problems while the standards were being drafted), Anger (when I realized the extent of the problem last year), Bargaining (lobbying FERC to order the standard be rewritten), Depression (which you can see in a number of my recent posts), and finally Acceptance (meaning I have come to realize that the more fundamental problems with the standard will never be addressed by someone with the authority to make their fix stick). 

However, if you work for a NERC entity, you probably envy me having the luxury of being able to “accept” the problems.  After all, I’m not on the hook for complying with CIP v5, but you are.  While I approach Nirvana in my blissful state of Acceptance, how are you supposed to keep out of the slammer on a charge of Aggravated CIP Violation?

This post started out as just another in my long series on the problems with CIP-002-5 (I’m sure I’ve done at least 30, probably more than that).  My subject this time is the word “programmable” in the NERC definition of Cyber Asset: “Programmable electronic devices”.  Basically, the problem is that “programmable” isn’t defined, and for some entities, the definition used could literally lead to additional millions in compliance costs.

A gentleman who I’ll discuss below told me how he is coping with this problem; it involves basically making up his own definition, documenting that, and making sure it is followed rigorously as his organization identifies Cyber Assets.  I will discuss all that, but I came to realize last week that what this person has done can be a paradigm for dealing with most of the unsolved problems in CIP-002-5.1 (and perhaps the other standards, although my guess is those problems are more easily addressed through normal “interpretation” activities than the CIP-002 problems are).

Yes folks, I’m saying you should consider the following procedure for dealing with the inconsistencies and vagueness in CIP-002-5.1:  Roll Your Own solution.  That’s why I’m calling this Part 1 in a series on how entities are making their own definitions, rewording the requirements so they make more sense, etc.  Of course, I’ll be pleased to hear how anyone else might be doing something similar (even if you won’t allow me to mention it in a post).  People have to be doing this now (perhaps inadvertently – in fact, I know many entities think they’re following the standard as written, when they’re probably not); otherwise, nobody would be making any progress at all on CIP v5.  But that’s OK; you need to move forward, even if that means Rolling Your Own.  And you’re in good company in any case.

And Now, Our Feature Story

I think a number of you are probably aware that the meaning of “programmable” is a big deal.  But I think that the people most aware of the problem are those in Generation, and especially owners/operators of large coal plants.

The problem is that these plants can have astounding numbers of electronic devices – transmitters, valve positioners, I/O cards, I/O processors, etc.  Someone who works for a large generation entity told me they have one plant that has 120,000 devices that could potentially meet the NERC definition of Cyber Asset. 

Let’s see, how long would it take to a) determine whether all of these were Cyber Assets or not; b) determine whether or not they were BES Cyber Assets; c) aggregate the BCA’s into BES Cyber Systems; and d) apply all of the requirements of CIP-003-6 (yes, 6) through CIP-011-2 (yes, 2) to these BCS?  My calculator won’t go that high, but my guess is it’s in the man-decades, if not man-centuries.  Something on the order of, “I guess we’ll have to stop generating electricity for a couple of years while everybody works on CIP version 5 compliance.”

And what makes all the difference?  The meaning of the word “programmable”.  Almost any electronic device can be “programmed” in some sense.  You might have to replace every bit of firmware in it, but by some definitions, this would constitute “programming” it.

I know there has been a lot of discussion in NERC circles about this problem, and I’m told it is one of the questions being addressed – in some way – by the new CIP Version 5 Transition Study Group (which I discussed in a recent post).  But entities really need to have this definition now (i.e. September 24, 2014 at 8:55PM Pacific Time), and many simply can’t wait any longer.

Enter the Hero.  This person is the CIP Compliance Manager for the Generation arm of a large IOU.  His job is to be compliant by April 1, 2016, not to tell everybody in his company to hold off doing any work on v5 until NERC does or doesn’t come up with a usable definition of “programmable”.  As he said in a recent email to me “Unless NERC defines a programmable logic device, the Responsible Entity should define it.”  There you go – the shot heard round the world.  Here is his definition of “programmable”:

An electronic device is normally considered as “programmable” if it has two or more of the following attributes:

a.        QWERTY or similar Keyboard or keyboard connection

b.        Management and/or programming port

c.        Serial (RS-232) or USB port

d.        Any routable protocol port, e.g. 10/100/1000BaseT port(s), 10/100/1000BaseFl port(s), WiFi, 802.x.

e.        Telephone line port

f.         Ability to configure three or more User accounts with complex passwords.

g.        Alterable memory, including firmware, that can be reprogrammed to change the function of the device.  This does not include alterable memory for the storage of configuration changes.

I won’t comment on this definition on a technical basis[i], but it does seem quite reasonable to me.  The bigger issue is, once you’ve decided on a definition[ii] for “programmable” – or any similar undefined term in the v5 standards, what do you need to do to feel comfortable actually applying it?

This gentleman did two primary things, which I recommend for anyone who wishes to follow in his footsteps (whether or not you use his definition of “programmable”, or even if this is for an entirely unrelated CIP v5 issue): He documented the definition and his rationale for wanting to use it, and he kept that as part of his compliance records.  He feels he's well prepared should an auditor challenge his definition three or four years from now.  And his answer will be simple - what else could I have done, given that there was no accepted definition to follow?

So that’s all I have to say.  There have really been two subjects to this post:

1. The definition of “programmable”
2. How at least one entity (and I know there are others doing the same thing) has decided to take matters into their own hands and develop their own definition. 

I’m implying – no, stating outright – that developing your own definitions and interpretations may be the only real “solution” to many if not most of the interpretation questions of CIP v5.  Of course, you need to follow whatever NERC and your region(s) say[iii] about these questions.  But you really can’t count on them to a) address every question, or b) provide definitive answers to the questions they do address.  You say this is unfair?  Here I quote – as I have before – the great philosopher Jimmy Carter: “Life is unfair.”

For the second post in this "Roll Your Own" series, go here.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Honeywell. 

---

[i] There is another blog that was recently brought to my attention – “CIPsecure” by Joseph Jimenez – that provided the following definition of “programmable” (you can find the post by scrolling to near the bottom.  It is dated Aug. 31):

A device that uses digital electronic technology to run pre-configured programming logic to interact with other devices over a communications interface. Programmable devices should also include a processing unit, memory, storage for the programming logic, and a communication interface that is remotely available. The remote communication interface should also allow for the access and modification of the stored programming logic.

At first glance, I think this is operationally not too different from my friend’s definition, shown later in my post.  By the way, Joseph’s blog seems excellent.  I intend to read all of his posts in the future.

Another “definition” can be found in a presentation that Felek Abbas of NERC gave to FRCC in May.  Here is his slide:

                        What makes something “programmable”?

                        * Contains firmware

                        * Firmware is modifiable via device interface (Ethernet, serial, parallel, USB, etc.)

                         

                        Configurable is not programmable.

                        * Electro-mechanical relays

                        * Dip switches

                        * If physical removal of chip is required to program device (EPROM etc.)

                         

My initial thought is that this definition will result in more devices being deemed “programmable” than either Joseph’s or my friend’s.  However, since this can’t be taken as an “official” NERC statement in any way, you should consider it no more than one person’s opinion.

[ii] Our Hero also pointed out to me that in the proposed NERC BES Cyber Asset Survey – which was recently cancelled – the question “How would you define programmable electronic device?” was asked.  Obviously, it would have been quite interesting to see the results for that question.

[iii] The CIP V5 Transition Study Group will post their documents here.