Building a 23 million record AI-driven database for council spending data

Identifying AI Opportunity

We needed to build a comprehensive dataset consolidating spending data from over 300 local councils. This information was publicly available but fragmented across multiple websites and formats, requiring significant manual effort to gather, standardise, and use effectively.

By carefully mapping their existing manual workflow, we identified significant opportunities to apply AI to automate the data extraction and standardisation process, making it vastly more efficient and usable.

Implementing AI Workflow

We developed an automated pipeline that systematically identified council spending URLs via targeted searches, extracted relevant data files (.csv/.xlsx), and leveraged AI to standardise inconsistent column headers into a unified database schema intelligently.

The solution incorporated automated data parsing, AI-driven mapping of diverse data formats, and integration with Companies House for supplier verification. We provided additional human oversight through selective data entry services to ensure completeness.

Outcome

• Successfully unified spending data from over 300 councils, comprising 23 million records, into a single, easily searchable database.

• Significantly reduced manual data processing time, improving both operational efficiency and data reliability.

• Enabled advanced filtering by council, supplier, and SIC codes, with streamlined Excel exports, creating new monetisation opportunities through premium features and data accessibility.