Unlocking the Black Box of UK Planning Data
For years, UK planning data has been technically public, but in practice, it’s been locked behind over 400 different council portals, each with its own unique schema and quirks. This has created a significant barrier to accessing and analyzing this critical data. However, one individual has spent four months scraping these portals, and the results are staggering – 2.6 million decisions across England, Scotland, and Wales from 241 councils.
The sheer complexity of the scraping problem cannot be overstated. With most UK councils running one of a handful of portal systems, such as Idox, it would seem like a straightforward task. However, each council has configured their system differently, with some blocking non-browser requests via TLS fingerprinting, and others imposing rate limits that can get you banned in under 10 minutes. This has necessitated the development of multiple scrapers, including a standard requests-based one, a Playwright-based one, and a curl_cffi one.
Despite these challenges, the data that has been scraped has yielded some fascinating insights. The approval rate for planning applications varies wildly by ward within a council, not just between councils. Furthermore, the time-to-decision data has revealed that 36.5% of home extension applications missed the statutory 8-week target in 2025, up from 27.9% in 2019. This has significant implications for the construction industry and homeowners alike.
The Approval Rate Paradox
While the national approval rate for planning applications is around 88%, this figure masks significant regional disparities. Some councils have approval rates that are substantially higher or lower than the national average. This raises important questions about the consistency and fairness of the planning process. Are some councils more lenient than others, or are there underlying factors at play that are driving these disparities?
Moreover, the fact that the approval rate varies so significantly by ward within a council suggests that local factors, such as the presence of conservation areas or the influence of local pressure groups, may be playing a significant role in shaping the planning process. This has important implications for developers and homeowners who need to navigate this complex landscape.
The approval rate paradox also raises questions about the role of data in the planning process. If data can reveal such significant disparities in approval rates, can it also be used to identify areas where the planning process is failing, and to develop more effective solutions? This is an area that warrants further investigation.
Who Wins and Who Loses in the Planning Game
The planning process is often characterized as a zero-sum game, where one person’s gain is another person’s loss. However, the data suggests that this is not always the case. In some areas, the planning process appears to be working relatively smoothly, with high approval rates and fast decision times. In other areas, however, the process is slow and contentious, with low approval rates and long decision times.
Developers and homeowners who are able to navigate the planning process effectively are likely to be the big winners in this game. They will be able to secure the permissions they need to build new homes and developments, while others will be left behind. On the other hand, those who are unable to navigate the planning process effectively will be the losers, facing delays, costs, and uncertainty.
However, the data also suggests that there may be a third group of winners and losers – the councils themselves. Councils that are able to streamline their planning processes and make decisions quickly and efficiently will be able to attract more developers and homeowners, and will be better placed to deliver the new homes and developments that their communities need.
The Skeptical Case
While the data has yielded some fascinating insights, it is also important to consider the skeptical case. What if the data is incomplete or inaccurate? What if the planning process is not as complex or contentious as the data suggests? What if the real problem is not the planning process itself, but rather the underlying economic and social factors that are driving the demand for new homes and developments?
Moreover, what if the solution to the planning problem is not more data or more efficient processes, but rather a more fundamental transformation of the way that we approach planning and development? This is an area that warrants further investigation and debate.
What’s Next for Planning Data
As the planning data landscape continues to evolve, there are several key milestones to watch. First, the development of more sophisticated data analytics tools will be critical in helping to unlock the insights that are hidden in the data. Second, the integration of appeal data into the planning data landscape will be essential in helping to identify areas where the planning process is failing, and to develop more effective solutions.
Finally, the use of data to drive policy change will be critical in helping to address the underlying economic and social factors that are driving the demand for new homes and developments. This is an area that will require close collaboration between policymakers, developers, and homeowners, and will require a deep understanding of the complex interplay between data, policy, and the built environment.
Pick one tactic from this post and apply it today. Which one will you start with?
By Daniel Cross, Digital Growth Strategist at TrendFlashy
Ready to launch your own asset?
Check out our guide on Building a Profitable Online Business.
