A key performance indicator(KPI) is a measure of performance, commonly used to help an organization defineand evaluate how successful it is, typically in terms of making progress towards its long-term organizational goals.
–KPIs provide business-level context to security-generated data
–KPIs answer the “so what?” question
–Each additional KPI indicates a step forward in program maturity
–None of these KPIs draw strictly from security data
Control Objectives for Information and Related Technology (COBIT) is a framework created by ISACA for information technology (IT) management and IT governance. It is a supporting toolset that allows managers to bridge the gap between control requirements, technical issues and business risks.
COBIT was first released in 1996; the current version, COBIT 5, was published in 2012. Its mission is “to research, develop, publish and promote an authoritative, up-to-date, international set of generally accepted information technology control objectives for day-to-day use by business managers, IT professionals and assurance professionals.”.
COBIT, initially an acronym for 'Control objectives for information and related technology', defines a set of generic processes to manage IT. Each process is defined together with process inputs and outputs, key process activities, process objectives, performance measures and an elementary maturity model. The framework supports governance of IT by defining and aligning business goals with IT goals and IT processes.
The framework provides good practices across a domain and process framework.
The business orientation of COBIT consists of linking business goals to IT goals, providing metrics and maturity models to measure their achievement, and identifying the associated responsibilities of business and IT process owners
COBIT 5 is the only business framework for the governance and management of enterprise IT. This evolutionary version incorporates the latest thinking in enterprise governance and management techniques, and provides globally accepted principles, practices, analytical tools and models to help increase the trust in, and value from, information systems. COBIT 5 builds and expands on COBIT 4.1 by integrating other major frameworks, standards and resources, including ISACA’s Val IT and Risk IT, Information Technology Infrastructure Library (ITIL®) and related standards from the International Organization for Standardization (ISO).
COBIT 5 helps enterprises of all sizes:
-Maintain high-quality information to support business decisions
-Achieve strategic goals and realize business benefits through the effective and innovative use of IT
-Achieve operational excellence through reliable, efficient application of technology
-Maintain IT-related risk at an acceptable level
-Optimize the cost of IT services and technology
-Support compliance with relevant laws, regulations, contractual agreements and policies
Figure 1. COBIT
The relationship between governance and management in COBIT is presented in the next figure.
Figure 2. Relationship between governance and management in COBIT 5(Courtesy to COBIT)
The governance and management key areas are presented below.
Figure 3. Governance and management key areas in COBIT 5((Courtesy to COBIT)
The relationship between KPI and COBIT is shown in the next figure.
Figure 4. Relationship between KPI and COBIT(Courtesy to COBIT)
The principles that are founding COBIT are presented in the next figure.
Figure 5. COBIT principles(Courtesy to COBIT)
How to choose KPI in relationship with vulnerability
KPI’s must be aligned at least with a business goal:
· KPI’s must be in accordance with risk or targets that are recognised by the various stakeholders. When the (business) risk or objective is not lucid a KPI will not come alive.
KPI’s are measured by measures. It is desirable to not define the KPI too specifically so as to avoid the KPI influencing the selection of measures still to be implemented.
Some questions could better define KPI.
· Are the KPI’s meant for internal use or for external communication to customers, partners, shareholders, affiliated organisations, certifying bodies, etc.?
Must they be connected to other standards in the sector so that KPI’s are somewhat comparable between organizations or is an organization free to choose their own definition?
Even when using the same framework (for example Cobit) the comparability of KPI’s will never be optimal. Organisations will always, in practice, make a translation to their own situation.
· How generic or how specific must a KPI’s definition be? Are they applicable to the whole company or just for a department?
In a centrally run organisation a KPI can be much more generically defined across n different process than in a de-centrally run organization. After all in a de-centrally organised organisation similar processes will differ more and therefore so will the measurability and steerage possibilities. And KPI’s must fit optimally with a process in order for it to be effective.
· Are KPI’s stable during the life cycle of the subject to be measurement?
During the life cycle of an application, for example, the need for measurements shifts.KPI’s that play a role in the development phase of applications differ from those in the operational phase. Also the target group being reported on can shift during the life cycle.Measurements must be trustworthy and meaningful
· What maturity level does an organisation or a process have and which KPI’s are useful for this maturity level?
Controlled measurement of certain KPI’s demands reproducibility and a tight framework in the execution of administration processes. KPI’ s meant for tuning of an administration process that is not (yet) there, have little additional value. The accumulation of detailed measurements and trend information concerning intrusions, for example, is of little use when an organisation does not have
its basic incident management process in order.· The measurability of the desired information is important.
The filling in of a security paragraph in a project phase document is simple to measure. Whether the quality of the paragraph is sufficient, is already more difficult to determine and without the involvement of security experts will probably not lead to accurate and objective information.
· In all scenarios the integrity of measurements is of great importance. The costs of the measuring must be in balance with the eventual objective. Often tooling is required to keep measurements cost-effective during a longer period of time, to guarantee integrity and to keep reports consistent. With manual aggregation of information, manipulation is possible and reliability cannot always be guaranteed.
-KPI’s on different levels must be correlated and provide a consistent view. Multiple
KPI on a lower level can be translated to one KPI on a higher level. Because these
KPI’s are usually geared towards another target group with a higher aggregation level,
word usage in the explanation must often also be adjusted. The word usage must fit in
with the actual world and interests of the target group.KPI must be easy to understand. Higher management may for example be more accountable for the percentage of application development projects where a risk analysis was carried out in due time, than for the completeness and depth
of an analysis carried out for a specific development project.
· Aside from standard reports there must be a mechanism for exception reports.Exceptions are important to keep people awake. In the case of exception reports it is often effective to give people ‘the naked truth’. Allow the shop floor to speak and do not try to make the reality prettier than it actually is. If for example, the management of a department does not tackle events that structurally occur and remains vague about it in his reports, it can be worthwhile to magnify an incident with
more than average impact. Diligence and care for disclosure of sensitive information remains an important factor.
· The chance that Security Management KPI’s are effectively used is greatest when applied to processes where people are already used to reporting other quality aspects by means of KPI’s.
This is reason to define Security Management KPI’s based on a process approach and to assure that KPI’s are built into processes where security is relevant, such as incident management and change management.
KPI design. KPI developed for vulnerability analysis
When an organisation wants to begin to define KPI’s it can generally choose two approaches,the top-down approach, which requires management commitment, and the bottom-up
approach whereby initial reports based on measurements at hand are the stimulant for more
and sharper reports.
An example of a bottom up approach is given below.
Determine security maturity e.g. by examining Stacey’s security model
Keep registration of:
· Incidents (reactive)
· Identified risks (proactive)
Classify these e.g. by sort, object, department.
Also think outside the borders
Personalise reports to target groups
For example: Legal matters, HRM, financial affairs
Aim is to increase awareness
Explaining reports per part area
- Point out consequences per responsibility area
Refer to damage table, in which damage categories are defined
· Formalise KPI’s: Where is the threshold?
· Ascertain trends
· Provide steerage by coupling trends to advised measures
We have defined the following KPIs for vulnerability:
Vulnerability level is given by other 5 lagging Kpi as seen in the figure.
Figure 6. Components of the Vulnerability level KPI
All the components are lagging KPI.
Figure 7. Vulnerability by operator
Figure 8.Vulnerability by task
A number of 5 specific KPI were defined as follows, based on the Capability Maturity Model (CMM):
Facts regarding WRT:
-Metric is best graphed
-Risk trend will decrease over time similar to 1/x
-Each defect criticality must have a non-linear factor assigned
•Critical = 10
•High = 5
•Medium = 2
•Low = 1
-Application business criticality must be rigidly defined
Facts regarding DRW:
–#1 most critical KPI
–DRW will be potentially very large at first
–Critical to shrink this metric as quickly as possible
–Can be used to target education where needed
–Important to note type of defect remediated (complex defects take longer to fix)
–Advanced defect tracking system
–Advanced web application security testing capabilities
–Capabilities to identify similar or likedefects across an application or logical trackable unit
Facts regarding RDR:
–Reappearing defects measure internal development confusion
–Recurring defects should prompt a systemic investigation into root-cause
–Critical for identifying poorly-run development organizations
–Advanced security testing capabilities using flow-based, data-driven methodology for completeness
–Integration with Quality Assurance for functional specification coverage
Facts regarding SCM:
–Most difficult KPI to achieve
–Most organizations cannot identify even known attack surface coverage
–Flow-driven & data-driven methodology is required to fully test known attack surface
–Exploratory testing required to discover “unknown functionality”
–Mature defect reporting system (tracking combined quality defects)
•Security as a quality defect
•Performance as a quality defect
•Functional (+related) as a quality defect
–Tight cooperation of Information Security & Quality Assurance
Facts regarding SQR:
–Final step in organizational maturity with respect to security testing
–Demonstrates security adoption as a component of overall software quality
 H.Bell and others (2006) GvIB Expert Letter – September 2006 ISSN 1872-4884, Volume 1 – No. 2
 Institute of Internal Auditors, UK&Ireland, Position Statement 2004, The Role of Internal Audit in Enterprise-wide Risk Management
 BS6079-3:2000 Project Management Part 3—Guide to the Management of Business Risk.
 Flesher, Dale (1996) Internal Auditing: A One-Semester Course, Florida: The Institute of Internal Auditors