Can You Manage Without Measures?

When looking at IT Strategy and Architecture it is very easy to become reactive. Each day the business throws up new challenges and directions often without consultation. From a security perspective the attackers always have the upper hand in knowing when and where they will attack while showing the business the value of proactive controls is difficult without hard data.

Crisis Management or Planned Chaos?

With the advent of Agile projects and an increased desire for operational efficiency it is easy to fall into a task focused responder role with strategy and architecture decisions becoming hastier or point in time. Exemptions are made on an ongoing basis for non strategy aligned projects because the business has progressed to a point that by the time the Architect discovers the project it is too late to align the solution towards the static strategy.

What are often intangible benefits make it hard to justify expenditure towards strategy and architecture especially when if both are done well then the worse case examples that would result in a loss will never be realized. Most organizations don’t have a strong actuarial database to record historic decisions and the financial impact over time. This makes showing how much an investment in a strategic direction has saved the business over time in productivity, availability, enablement or security. It also fails to record the impact of any losses due to oversight of project or delivered risks.

An increasing trend towards using risk based security for IT projects and the subjectivity around what is and is not a risky activity makes it easy for the organization to drift too far from each side of the threshold. On the one hand you typically have the project manager who is focused on delivering a project on time, on budget and (hopefully) within scope. On the other you have the security architect & risk adviser who is focused on what are often subjective measures of minimizing risk.

All of these influences combined with an “as a service” culture where the IT department looks to internal users & consumers as valued customers and are subsequently evaluated on providing excellent customer service the use of Key Performance Indicators (KPI) becomes increasingly important.


Real Time Data

When looking at how to measure IT strategy and architecture much can be learned from operations and even the sales department. There is little value in discovering that you have missed your measures at the year-end when performance evaluations are due and bonuses are being allocated. In the same way that the server team likes to know proactively that a server disk is about to fail or a sales manager doesn’t want to have to step in at the last week of the quarter to close some sales to hit the revenue targets. IT strategy and architecture measures need to flow through in real time or as near real time as possible.

Outliers or Black Swan events can tip the scales quickly and drastically in either direction. Finding out that a key project has significant unmitigated risks the day before go live (generally at the go live meeting with the executive team) can quickly result in poor KPI values with little opportunity to restore or direct a project in a more aligned direction.

As a result when looking at measures try and collect data in real time or as near real time as possible. Rather than relying solely on month, quarter, half year or full year values.


What to Measure

Determining what to measure will be different for each business. Each measure however needs to be traceable upwards to a strategic imperative and each strategy item should have at least one measure. The purpose of each measure should be questioned, along with whether the data is the best or most suitable source for collection. Measures should highlight gaps in execution of the strategy early enough to enable restoration but should also focus on areas that have a potential for overinvestment. Risk appetite measures should focus on aggregated and single events that would result in the business falling either outside of appetite or leaning too far inside resulting in the loss of opportunity as a result of risk aversion. Finally when looking at measures ensure that all organization dimensions are evaluated including people and culture to ensure both a short and long term view is obtained.


How to Measure

Use a combination of leading and lagging measures and automate collection where possible. Often portal or CRM types of technology can provide a good base of data collection, manipulation and presentation and should be accessible to most organizations. More sophisticated measuring software is available such as risk and service management software that can also be used if available. The key objective is to get the data sooner and with less effort.

Look towards how you can publish data internally and show trending towards targets. Building data over time can show progress and highlight value of strategy and architecture to the business. Additionally where numbers are lower than desired it clearly communicates to the team areas of focus and sets a clear expectation of desired performance with no year end surprises.

Importantly look for early warning indicators. Perform post mortems on good and bad projects where strategy and architecture worked well and not so well. Looking at root cause for both situations across a number of projects will highlight measures that should predict performance.

Be careful of overdoing measures; focus on key and significant measures rather than a vast quantity of less relevant information. Often near enough is close enough especially when looking at leading indicators. Having a full data set is valuable but consideration should be given to the subsequent delays of not being able to act until all information is on hand. Look towards using Time Based Activity Costing (TBAC) as a short cut for productivity measures and build data out over time to assist with future forecasting.



When looking toward leading measures balance the quality of data versus the time taken to collect. Work out what data can be simplified to provide a suitable but not necessary perfect measure.

Likewise be careful placing too much priority on certain measures over other measures. This can have undesirable results. For example a measure of how many times an Architect got a solution right the first time with no rework could result in much longer working times, longer than multiple iterations of the same solution as a result of rework.

Remember that remuneration dictates behavior. Tying measures to performance can result in employees trying to manipulate the values of measures. Not only will this hide the true data but can lead to short term decision making based on the focus on specific measures. Think carefully about what sort of behavior specific measures will dictate.


Further Resources

Kaplan & Norton – Using the Balanced Scorecard as a Strategic Management System

SABSA Lifecycle -

Kelvin Rundle

Roxsolt Pty Ltd, Sydney, Australia

Team owner