VULNERABILITY METRICS AND KPI

KPI definition

A key performance indicator(KPI) is a measure of performance, commonly used to help an organization defineand evaluate how successful it is, typically in terms of making progress towards its long-term organizational goals.

–KPIs provide business-level context to security-generated data

–KPIs answer the “so what?” question

–Each additional KPI indicates a step forward in program maturity

–None of these KPIs draw strictly from security data

COBIT

Control Objectives for Information and Related Technology (COBIT) is a framework created by ISACA for information technology (IT) management and IT governance. It is a supporting toolset that allows managers to bridge the gap between control requirements, technical issues and business risks.

COBIT was first released in 1996; the current version, COBIT 5, was published in 2012. Its mission is “to research, develop, publish and promote an authoritative, up-to-date, international set of generally accepted information technology control objectives for day-to-day use by business managers, IT professionals and assurance professionals.”.

COBIT, initially an acronym for 'Control objectives for information and related technology', defines a set of generic processes to manage IT. Each process is defined together with process inputs and outputs, key process activities, process objectives, performance measures and an elementary maturity model. The framework supports governance of IT by defining and aligning business goals with IT goals and IT processes.

The framework provides good practices across a domain and process framework.

The business orientation of COBIT consists of linking business goals to IT goals, providing metrics and maturity models to measure their achievement, and identifying the associated responsibilities of business and IT process owners

COBIT 5 is the only business framework for the governance and management of enterprise IT. This evolutionary version incorporates the latest thinking in enterprise governance and management techniques, and provides globally accepted principles, practices, analytical tools and models to help increase the trust in, and value from, information systems. COBIT 5 builds and expands on COBIT 4.1 by integrating other major frameworks, standards and resources, including ISACA’s Val IT and Risk IT, Information Technology Infrastructure Library (ITIL®) and related standards from the International Organization for Standardization (ISO).

Benefits

COBIT 5 helps enterprises of all sizes:

-Maintain high-quality information to support business decisions

-Achieve strategic goals and realize business benefits through the effective and innovative use of IT

-Achieve operational excellence through reliable, efficient application of technology

-Maintain IT-related risk at an acceptable level

-Optimize the cost of IT services and technology

-Support compliance with relevant laws, regulations, contractual agreements and policies

Figure 1. COBIT

The relationship between governance and management in COBIT is presented in the next figure.

Figure 2. Relationship between governance and management in COBIT 5(Courtesy to COBIT)

The governance and management key areas are presented below.

Figure 3. Governance and management key areas in COBIT 5((Courtesy to COBIT)

The relationship between KPI and COBIT is shown in the next figure.

Figure 4. Relationship between KPI and COBIT(Courtesy to COBIT)

The principles that are founding COBIT are presented in the next figure.

Figure 5. COBIT principles(Courtesy to COBIT)

How to choose KPI in relationship with vulnerability

KPI’s must be aligned at least with a business goal:

· KPI’s must be in accordance with risk or targets  that are recognised by the various stakeholders[1]. When the (business) risk or objective is not lucid a KPI will not come alive.

KPI’s are measured by measures. It is desirable to not define the KPI too specifically so as to avoid the KPI influencing the selection of measures still to be implemented.

Some questions could better define KPI.

· Are the KPI’s meant for internal use or for external communication to customers, partners, shareholders, affiliated organisations, certifying bodies, etc.? 

Must they be connected to other standards in the sector so that KPI’s are somewhat comparable between organizations or is an organization free to choose their own definition?

Even when using the same framework (for example Cobit) the comparability of KPI’s will never be optimal. Organisations will always, in practice, make a translation to their own situation.

· How generic or how specific must a KPI’s definition be? Are they applicable to the whole company or just for a department?

In a centrally run organisation a KPI can be much more generically defined across n different process than in a de-centrally run organization. After all in a de-centrally organised organisation similar processes will differ more and therefore so will the measurability and steerage possibilities. And KPI’s must fit optimally with a process in order for it to be effective.

· Are KPI’s stable during the life cycle of the subject to be measurement?

During the life cycle of an application, for example, the need for measurements shifts.KPI’s that play a role in the development phase of applications differ from those in the operational phase. Also the target group being reported on can shift during the life cycle.Measurements must be trustworthy and meaningful

· What maturity level does an organisation or a process have and which KPI’s are useful for this maturity level?

Controlled measurement of certain KPI’s demands reproducibility and a tight framework  in the execution of administration processes. KPI’ s meant for tuning of an administration process that is not (yet) there, have little additional value. The accumulation of detailed measurements and trend information concerning intrusions, for example, is of little use when an organisation does not have

its basic incident management process in order.· The measurability of the desired information is important.

The filling in of a security paragraph in a project phase document is simple to measure. Whether the quality of the paragraph is sufficient, is already more difficult to determine and without the involvement of security experts will probably not lead to accurate and objective information.

· In all scenarios the integrity of measurements is of great importance. The  costs of the measuring must be in balance with the eventual objective. Often tooling is required to keep measurements cost-effective during a longer period of time, to guarantee integrity and to keep reports consistent. With manual aggregation of information, manipulation is possible and reliability cannot always be guaranteed.

-KPI’s on different levels must be correlated and provide a consistent view. Multiple

KPI on a lower level can be translated to one KPI on a higher level. Because these

KPI’s are usually geared towards another target group with a higher aggregation level,

word usage in the explanation must often also be adjusted. The word usage must fit in

with the actual world and interests of the target group.KPI must be easy to understand. Higher management may for example be more accountable for the percentage of application development projects where a risk analysis was carried out in due time, than for the completeness and depth

of an analysis carried out for a specific development project.

· Aside from standard reports there must be a mechanism for exception reports.Exceptions are important to keep people awake. In the case of exception reports it is often effective to give people ‘the naked truth’. Allow the shop floor to speak and do not try to make the reality prettier than it actually is. If for example, the management of a department does not tackle events that structurally occur and remains vague about it in his reports, it can be worthwhile to magnify an incident with

more than average impact. Diligence and care for disclosure of sensitive information remains an important factor.

· The chance that Security Management KPI’s are effectively used is greatest when applied to processes where people are already used to reporting other quality aspects by means of KPI’s.

This is reason to define Security Management KPI’s based on a process approach and to assure that KPI’s are built into processes where security is relevant, such as incident management and change management.

KPI design. KPI developed for vulnerability analysis

 When an organisation wants to begin to define KPI’s it can generally choose two approaches,the top-down approach, which requires management commitment, and the bottom-up

approach whereby initial reports based on measurements at hand are the stimulant for more

and sharper reports.

Top-down

The basis and starting point in the model is obtaining a strong top-level management

commitment. Subsequent policies and measurements must be defined.

The top-down approach will work especially when management is experiencing great pressure, like for complying with SOX regulation on time.This is only attainable in the short run with

sufficient management commitment.This approach will, in nearly all cases, be based on checklists from existing models like Cobit.

Bottom-up

An alternative approach is to ‘just start anywhere’ with presentation of easily obtainable

measuring results from operational processes. In this way management is made aware of the

interesting information reported from processes which can be used for steerage. Initially the

information offered will not suitably comply with the information requirement of the process

owner, but the imagination of the process owner will be stimulated and he will be in a better

position to define a modified KPI which is more in line with the requirement.

By means of an iterative process, KPI’s can be constantly improved. Important in this

bottom-up approach is that expectations must be well managed to avoid the threat of a

process owner quitting prematurely and losing his confidence.

An example of a bottom up approach is given below.

No	Actions
1	Determine security maturity e.g. by examining Stacey’s security model
2	Keep registration of: · Incidents (reactive) · Identified risks (proactive) Classify these e.g. by sort, object, department. Also think outside the borders
3	Personalise reports to target groups For example: Legal matters, HRM, financial affairs Aim is to increase awareness
4	Explaining reports per part area - Point out consequences per responsibility area Refer to damage table, in which damage categories are defined
5	Expand this: · Formalise KPI’s: Where is the threshold? · Ascertain trends · Provide steerage by coupling trends to advised measures

We have defined the following KPIs for vulnerability:

Vulnerability level (V_l)- Vulnerability level expresses the overall vulnerability for the whole company or for the section analysed. Vulnerability level could be measured on a scale from 1- no significant vulnerability to 5- extreme vulnerability. Vulnerability level is a lagging KPI.

V_l=(V_bO+V_bOP+V_bM+V_bT+V_bE)/5

The oposite of V_l would  express the level of preparedness against the action of vulnerabilities.  The KPI corresponding is:

Preparedness level (P_l)= 1/V_l  Preparedness level is a leading KPI.      

Vulnerability level is given by other 5 lagging Kpi as seen in the figure.

Figure 6. Components of the Vulnerability level KPI

All the components are lagging KPI.

Vulnerability by Outsider (V_bO)- gives the enterprise vulnerability to the attack by outsiders. Such attacks could be direct (physicall) or could be virtual, through the network of the enterprise, towards the command and control functions, etc. The more the enterprise is interconnected in a large technical and societal network ,the more it is exposed to attacks by the outsiders. There are two main kind of  outsider attackers:

-with financial motivation- are looking for financial awards as a result of the attack;

-with other motivation such as revenge, curiosity, etc.

Financial motivated attackers could be more easy identified and their attack prevented. The most valuable assets of the enterprise must be guarded and proteged . Redundant solutions should be ready in the case of failure of critical significant systems.

Vulnerability by Operator (V_bOP)-describes the vulnerabilities introduced in the system by the employees. The operator could be trained insufficiently or incorrectly, he could make mistakes because its physical and psychical  state, could be overstressed or affected by events that are not connected with the work. On the other side , the operator could make intentionally mistakes, for various reasons, from financial ones (for example he introduces a speed of work at machine X greater than the prescribed parameter in order to gain more money) till the ones connected with his relationship with the management.

Figure shows the main elements that are leading to vulnerability by operator.

Figure 7. Vulnerability by operator

Vulnerability by Machine (V_bM) is given by the defects and failures that could appear during the work process at the machines and instalations that are used in this process.It could be written that:

V_bM=f(Complexity, Protection, Work regime, Capability of repair)

Vulnerability by machine[2] is a function of complexity of the machine (the greater complexity leads to a greater vulnerability to fail), existing protection systems, the work regime in which the machine performs (better performances are requiring a more intensive work regime which could overpass the prescribed thresholds) and also of the training of the operator and the capability of repairing the machine in work.

Vulnerability by task (V_bT) is specific to the task being developed, transmitted and processed by the operator.Figure shows the components for this type of vulnerability

Figure 8.Vulnerability by task

Vulnerability by Environment(V_bE) is given by the environment influnece on the work and on the global vulnerability. If the work is done outside or in closed spaces or in noxes that are not properly captured, etc. it could lead to loss or even incidents and work accidents. For example a bad insulated ceiling could lead to a waterfall over the electric instalation. Work in open spaces could lead to health problems, etc.

A number of 5 specific  KPI were defined as follows, based on the Capability Maturity Model (CMM):

WRT–Weighted Risk Trend=A business-based representation of risk from vetted (web) application security defects over a specified time-period, or repeated iterations of application development-Maturity level(rank): 1

WRT=[(Multiplier_criticalx defects) + (Multiplier_highx defects) + (Multiplier_mediumx defects) + (Multiplier_lowx defects)] x *Criticality_business

Facts regarding WRT:

-Metric is best graphed

-Risk trend will decrease over time similar to 1/x

-Each defect criticality must have a non-linear factor assigned

•Critical = 10

•High = 5

•Medium = 2

•Low = 1

-Application business criticality must be rigidly defined[3]

•Business critical

•Critical

•Important

DRW–Defect Remediation Window=The length of time from when a vetted (web) application security defect is identified until it is verified closed-Maturity level:2

Facts regarding DRW:

–#1 most critical KPI

–DRW will be potentially very large at first

–Critical to shrink this metric as quickly as possible

–Can be used to target education where needed

–Important to note type of defect remediated (complex defects take longer to fix)

RDR–Rate of Defect Recurrence=The rate, over time, at which previously closed (web) application security defects are re-introduced into a given application, organization, or other logical unit-Maturity level:3.

Requirements:

–Advanced defect tracking system

–Advanced web application security testing capabilities

–Capabilities to identify similar or likedefects across an application or logical trackable unit

Facts regarding RDR:

–Reappearing defects measure internal development confusion

–Recurring defects should prompt a systemic investigation into root-cause

–Critical for identifying poorly-run development organizations

SCM–Specific Coverage Metric=The flow-based or component-based coverage of total functionality that web application security testing has achieved-Maturity level:4

Total functionality = known functionality + discovered functionality

 Requirements

–Advanced security testing capabilities using flow-based, data-driven methodology for completeness

–Integration with Quality Assurance for functional specification coverage

Facts regarding SCM:

–Most difficult KPI to achieve

–Most organizations cannot identify even known attack surface coverage

–Flow-driven & data-driven methodology is required to fully test known attack surface

–Exploratory testing required to discover “unknown functionality”

SQR–Security to Quality defect Ratio[4]=The ratio of security defects to the total number of software quality defects being generated (functional + performance + security)-Maturity level:4.

SQR=D_s/D_t: Ds= Total Security defects; Dt= Total Overall Quality defects

Requirements

–Mature defect reporting system (tracking combined quality defects)

•Security as a quality defect

•Performance as a quality defect

•Functional (+related) as a quality defect

–Tight cooperation of Information Security & Quality Assurance

Facts regarding SQR:

–Final step in organizational maturity with respect to security testing

–Demonstrates security adoption as a component of overall software quality

---

[1] H.Bell and others (2006) GvIB Expert Letter – September 2006 ISSN 1872-4884, Volume 1 – No. 2

[2] Institute of Internal Auditors, UK&Ireland, Position Statement 2004, The Role of Internal Audit in Enterprise-wide Risk Management

[3] BS6079-3:2000 Project Management Part 3—Guide to the Management of Business Risk.

[4] Flesher, Dale (1996) Internal Auditing: A One-Semester Course, Florida: The Institute of Internal Auditors