Does size matter? Getting to grips with big data in local government

Local authorities have a huge opportunity to deliver savings and improve citizen experience by mining the gold in the data that they already hold.  Jonathan Bradshaw, director of technology at Agilisys, offers his advice on finding the value in your information assets.  


The digital data universe is big and is getting bigger by the day.  The many cameras, sensors and billions of people socialising online are likely to balloon the nine zettabytes (which is nine trillion gigabytes – and to give you some idea, the standard laptop holds 500 gigabytes of storage) of digital data in the world today to almost 50 zettabytes by 2020.  In parallel the stream of innovations helping to make sense of all this data continues to grow ever stronger.  And there is a stack-ful of technology that can be obtained for free, thanks to the miracle of open source software such as Apache Hadoop and its ecosystem of Pig, Hive, Spark, Oozie and the rest.

Better still is that the big cloud providers such as Amazon, IBM and Microsoft are not only making it amazingly cheap to host and process massive data sets, but that they themselves are innovating with tools to combine, analyse, visualise and generally drive valuable insights from big data.

All this technology can sometimes be a distraction for a local authority facing cost and service challenges as not all data that they hold is considered ‘big’.  Big data is categorised when it can neither be managed nor analysed by ‘traditional’ methods and carries the ‘three Vs’: volume, variety and velocity.


Shining examples  

Recent examples of local authorities driving value from existing transactional data sets include the London Borough of Camden, who delivered savings by combining data sets from transactional systems to prevent tenancy fraud.  Another local authority increased foster parent applications by targeting those with the greatest propensity to volunteer. Analytics of existing data sets have been the key enabler behind the forecast counter-fraud savings of £7m at Ealing and £10m at Birmingham City Council, which helped provide motivation for the Government’s award of £16m to over 30 councils to tackle what the Annual Fraud Indicator reports is the £2.2bn annual cost of fraud in local government.

However, the biggest barrier that remains for local governments trying to get more value out of the data already held is often a cultural challenge rather than a technological one; most of the useful data is held or ‘owned’ within council systems and departments. As well as sharing between public sector organisations and agencies, council departments themselves need to break silos to share what has traditionally been considered ‘their’ data.

Some councils have attempted to overcome this by developing data insight teams that span departments. Some have drawn inspiration from the Mayor’s Office of Data Analytics (MODA) that Mayor Bloomberg Created in New York.  A study on MODA, published last summer, concluded that getting value out of public data is not about building the technology solution of the future with sensors on every street but about leadership and starting to make progress now with the data that has already been collected.


A revolutionary role 

The role of Chief Data Officer (CDO) is now increasingly seen in private sector organisations with board level responsibility for the governance and use of data as a business asset.  The role is increasingly appearing in public sector organisations with CDO’s appointed in the NHS England and the UK Government; important signs that the public sector is recognising the value of data.

Positive strides and changes in legislation, such as the local government transparency code and Tim Berners-Lee co-founded Open Data Institute have further encouraged local authorities to share data with each other and the public they serve.  Initiatives such as these have helped to create the environment in which over 25,000 data sets are now published on  Making programmable data accessible over APIs has helped to spawn a wave of innovation such as the popular Citymapper app built on TfL data sources.  But there is still more that can be done.

In a world of reducing budgets, local authorities have yet to understand that better and cheaper services can be obtained by combining public and other external data sources with their own internal data. The time of the local authority CDO is now upon us. The prize of embracing a ‘digital by default’ approach will not be fully grasped if something good doesn’t come of the data it generates.

Jonathan Rosenberg, the former Senior Vice President of Products at Google, arguably the biggest data driven company in the world today, beautifully captured the importance of data: “Data is the sword of the 21st century; those who wield it well, the Samurai.”  The time is ripe for those authorities without a strategy to build a data driven organisation to bare their sword and start their journey to become data warriors.


Agilisys works with local and central government to deliver transformative IT and customer services delivering real benefits for citizens across the UK.

People are at the heart of our innovative approach, ensuring that our digital first philosophy delivers tangible solutions for the real issues facing local and central government. Agilisys also leads the way in pioneering new business models, recently becoming an employee owned organisation, which means the people working in the organisation are passionately looking for new ways to add value.

The Agilisys suite of market-leading technology platforms and experienced consultants already help over 60 local authority partners transform their service levels and deliver significant and lasting improvements to their performance and efficiency.

To find out more follow Agilisys on Twitter at @Agilisys or visit


Related reading