Permissions and performance improvements in Wagtail 5.1
Exploring the recent (re)work of our permissions system in Wagtail 5.1 and the impact it brought to Wagtail's performance
When building websites using a content management system (CMS) like Wagtail, it is common to have multiple teams with different access levels working on different content. To control which user has access to what content, Wagtail adapts and extends Django's permission system.
As a CMS, Wagtail works with different types of content: pages, images, documents, snippets, and anything else that is registered in the admin. Each type of content has its own characteristics. For example, pages in Wagtail have a distinct characteristic of being organized in a tree-based structure. These characteristics need to be taken into account when implementing Wagtail's permission system.
Wagtail started off by focusing on the management of content as pages. Over the years, Wagtail has evolved so much and the permission system became increasingly complex as new features were added. Some of these features are (or were) exclusive to pages, so we added a few parts in the permission system to handle and optimize page-specific cases.
Unfortunately, there were parts within Wagtail where these optimizations were not used effectively. This led to duplicated database queries that could have been avoided.
We reworked the permissions system in Wagtail 5.1 and saw up to a 65% reduction in the number of database queries, which made the admin faster in all scenarios we tested. This is the story of how we achieved that.
The UserPagePermissionsProxy class
If you read the upgrade considerations in Wagtail 5.1, you might notice that we deprecated the UserPagePermissionsProxy class. This class may be familiar to those of you who have heavily customized your Wagtail instance or dug into Wagtail's codebase. But what is it exactly, and why are we dropping it? 🤔
Before we get into that, I'll briefly explain how the Wagtail's permission system works for pages.
Group, page, permission
Wagtail's page permission system is attached to Django's "user groups" mechanism and the "tree" structure of pages. In short, you can create a "mapping" between:
- A group of users
- A particular page
- A permission (e.g., edit or publish)
Then the same permission will also apply to all its child pages and further descendants.
For example, if you create a mapping between the "moderators" group, the "blog index" page, and the "publish" permission, then the "moderators" group will also have "publish" permission for all "blog post" pages.
You can make as many mappings as you like. If you want to add the "lock" permission as well, then you can create another mapping with the "lock" permission for the same group and page.
This is configured through the "Groups" edit view in the Wagtail admin. The screenshot below shows the edit view for a group's page permissions, and each "mapping" is represented by a checkbox.
If a page has no specific permission mapping, you can look at the page's ancestors to figure out the permissions. By default, permissions are assigned on the "root" page, which means they apply to all pages in Wagtail.
With this information, if you know all the permission mappings that are assigned to the groups a particular user is in, then you can infer the user's permissions for any given page. Now, let's go back to the UserPagePermissionsProxy class.
Suboptimal optimisation
The idea for the UserPagePermissionsProxy class is that when this class is instantiated, it runs a query to get all the permission mappings for a user. The instance is then cached in a template context variable to be reused anywhere in the rendered template in a single request-response cycle. This means that a single query is sufficient to work out any page permissions-related checks for any Wagtail page object when you load a view in the admin. Or at least, that was the idea.
Here is an example: When you're browsing the page explorer in the Wagtail admin, a single load of the explorer view requires permission checks for each page shown in the listing. The checks are needed to show what actions you can do because you can have different permissions for each of the pages. By querying all the permission mappings in one go and caching them, we only need to run a single query instead of one for each page (similar to the N+1 query problem).
For the most part, the UserPagePermissionsProxy class did accomplish those optimizations. However, in a single render of a Wagtail admin view, there are other parts where permission checks need to be done and the Python code does not (and should not) have access to the template context.
The main menu sidebar is one example of this: the items in the menu are managed by the server's Python code, but they are rendered on the client using React. Another example is reusable code such as the code for copying or unpublishing a page, which can be executed without the context of rendering an admin view.
In those cases, we ended up having to create a new UserPagePermissionsProxy instance, re-running the same query that was already done someplace else. That's not good 😢
Investigating further, it turned out that this problem was not isolated to pages. Remember when we said that Wagtail's permission system has to consider the characteristics of different types of content? Well, about that...
Permission policies
The UserPagePermissionsProxy class is specific to pages and has always been there since Wagtail was created. As Wagtail grew and added support for managing other content, such as images and documents, we implemented a more generic approach to permissions: "permission policies". This was added back in 2016.
A permission policy is a class that gives you the ability to make permission-informed queries about a particular type of content, such as "does this user have permission to add images", "which users can edit this image", "which documents can this user edit", etc. All permission policies follow (and extend) a standard interface for most common use cases defined by the base permission policy class. It's basically a more standardized version of UserPagePermissionsProxy.
Permission policies are not cached in the template context, but they also did not have any caching at all. In the end, they suffered the same problem we had with UserPagePermissionsProxy.
In an ideal world, we would have a more standardized approach for permissions, regardless of whether it's for pages or other types of content. A more unified permission system would allow us to introduce better permissions caching that is applied across all types of content.
Such a permission system could also lead to more advanced permission customization by developers, something that has been requested for years. We knew it wouldn't be an easy feat to accomplish because we wanted to optimize and stabilize the system before making it an officially documented part of Wagtail. Otherwise, developers would have to rework their customizations as we change the implementation details.
So, what could we do about it?
A better caching mechanism
Some time ago, we received a contribution from our Wagtail friend Tidiane that could reduce the queries executed by the UserPagePermissionsProxy object. The reduction was accomplished by moving the caching from the template context to the user object. This contribution showed some very promising performance improvements, as you can see in the PR description.
You might wonder why the PR was not merged. Looking at the changes involved, there were some advanced Python techniques and possible breaking changes to users who might have made some deep customizations. The changes were quite risky, so the core team didn't merge it into the codebase.
That said, the idea to move the caching to the user object was brilliant. Permission checks are always concerned about a particular user (or users), so it made sense to store the cache closer to the user. Wherever you need to do a permission check, it's very likely you have access to the user object in the code. So if the cache is accessible from there, surely we could make use of it!
Fun fact: if you're familiar with how Django internally does basic permission checks, it also does some caching on the user object. So, this approach isn't totally new or too "out there"!
Okay, so we found a better approach for caching the permissions. What did we do next?
Flying two birds with one rocket
Permission policies were implemented after the idea for a standardised permission system within Wagtail came up. At that time, UserPagePermissionsProxy was already in place and used throughout Wagtail, so we couldn't implement a "page permission policy" immediately. To do so would require internal rework within various parts of Wagtail so that UserPagePermissionsProxy is not used anymore, or at least so that it uses permission policy under the hood.
The future plan for moving towards permission policies is also why we were hesitant to make further changes to UserPagePermissionsProxy. This is the reason why we had to reject a PR to allow UserPagePermissionsProxy to be customizable as well as Tidiane's PR to optimize UserPagePermissionsProxy.
Well, once we identified a few problems within the permission system and a potentially unified approach to fix them, we decided that there was never a better time to do some (re)work! We roughly broke the work down into these steps:
- Implemented a PagePermissionPolicy class to ensure we have an optimized set of methods to do permissions-related page queries that follow our standardized permission policy interface.
- Refactored UserPagePermissionsProxy to use PagePermissionPolicy so that existing permission checks made use of the optimized methods under the hood without changing how the checks are performed on the surface level.
- Deprecated and removed all internal uses of UserPagePermissionsProxy to prove that we no longer need UserPagePermissionsProxy, and made sure developers who use the class in their projects would still have time to migrate before it is fully removed from Wagtail.
- Optimized the existing collections permission policies using the same approach we introduced in PagePermissionPolicy so that permission checks for images and documents were also optimized.
- Refactored the GroupPagePermission model (which is used to store the permission mappings described earlier) to use Django's Permission model instead of a plain string so that it aligned better with the GroupCollectionPermission model used for collections.
- Fixed a few issues in collections permission policies that mostly had gone unnoticed.
Measuring the impact
Now that we did all that work, let's see how those changes affected Wagtail.
The addition of PagePermissionPolicy "completed" our permission policies module as the underlying foundation of permission queries in Wagtail. While it is not exactly a quantifiable improvement, we are certain that this module will streamline Wagtail's development in areas where permissions are concerned. We've seen some improvement in the subsequent cleanups we did to prepare for the upcoming Universal Listings feature.
Meanwhile, the optimizations we made to the permission policies are definitely something that we can measure in numbers. To compare Wagtail's performance before and after the optimizations, we used Django Debug Toolbar, which is a well-known tool to debug and analyze the performance of a Django project. To get reproducible results, we ran the tool against a fresh instance of our bakerydemo project.
After logging into the Wagtail admin using an account that's part of the Moderators group on the bakerydemo, we got the following results for the dashboard view on Wagtail 5.0.2.
The summary shows that we had 63 queries, including 41 similar and 39 duplicates. Among those queries:
- A query to the "wagtailcore_grouppagepermission" table was duplicated three times
- A query to the "wagtailcore_groupcollectionpermission" table was duplicated three times
- A query to the "wagtailcore_page" table with a join to the "wagtailcore_grouppagepermission" table was duplicated five times.
There were more similar and/or duplicated queries further down below, which didn't look good.
After upgrading Wagtail 5.1 and re-running the debug toolbar, we got the following results.
The summary shows that we had 23 queries and it doesn't show us anything about similar or duplicate queries. Looking at the details, you can see we no longer have such inefficient queries! There is only one query to the "wagtailcore_grouppagepermission" table and only one query to the "wagtailcore_groupcollectionpermission" table. Our optimizations worked!
And this is just for the dashboard view. Wagtail does permission checks in most (if not all) views throughout the admin, so we decided to investigate if we would see similar improvements across the board.
To make benchmarking easier and reproducible, I wrote some automated browser tests using Playwright to run the debug toolbar and report its SQL summaries.
The automation spins up the bakerydemo with a fresh database and logs into the Wagtail admin as three different users: a superuser, an editor, and a moderator. Once logged in, some of the admin views are loaded, e.g. the dashboard, the page explorer, and the page editor views. The debug toolbar output for each view is then recorded and reported when the tests finish.
In numbers
After running the benchmark against Wagtail 5.0.2, we got the following results.
According to the report, we had a lot of similar and duplicated queries for each view. If you look at the superuser cases, the queries tended to be lower. In the permission system, we skipped database queries for superusers because we can assume they always have permission. The fact that there are more similar and duplicated queries for non-superusers meant the problem was likely related to permissions.
Meanwhile, here are the results we got for Wagtail 5.1.
The report shows that the similar and duplicated queries went down significantly in almost all cases — even to 0 for some of them. The overall number of queries went down by 15-65% across all the scenarios we tested, which led to faster query times. At a higher level, you can see that even the benchmark itself ran more than 20 seconds faster, from ~105 seconds to ~83 seconds.
For more details on the results, you can check out the spreadsheet and charts that we compiled. Or, you can also run the benchmark yourself! A wise man once said, "talk is cheap, show me the code". So, I made a branch on my bakerydemo fork which you can use to run the same benchmark.
In conclusion
The permission system in Wagtail needed some long-overdue refactoring and optimizations. In Wagtail 5.1, we made some improvements in this area by reducing the database queries through a unified and optimized interface. These changes resulted in a significant performance improvement across the Wagtail admin and opens up the possibility of making the permissions system more customizable in the future.
