[Wikipedia-l] Comprehensive template expansion size limit

[Tim Starling]

I have in my working copy a set of changes introducing a comprehensive 
template expansion size limit. The limit is configurable (by 
$wgMaxArticleSize), and will probably be set to 1 MB on Wikimedia. This 
prevents a number of DoS vulnerabilities, such as the exponential argument 
substitution attack that we've known about for some time.

The output size of template expansion has long been limited to 1MB, limiting 
several attacks. But it was possible to evade this limit by forcing the 
parser to parse a large chunk of text, and then throwing away the result, 
say by putting it into an unused template parameter. The new behaviour is to 
limit both the pre-expansion size and the post-expansion size.

This may break some articles, that's why I'm posting this to wikipedia-l. In 
particular, it will break articles which use {{#switch}} or similar 
constructs to extract many small fragments of text from large 
pseudo-databases. This practice is extremely inefficient, and I'd very much 
like to see it discontinued on the editorial level. This feature will 
probably only break the most flagrant cases, where article parse times were 
in the tens of seconds. I'm sure the other developers would agree -- we 
would be very grateful if the remaining cases could be dealt with by policy, 
as much as possible. I've been having a discussion with some Wikipedians 
along these lines at:

http://en.wikipedia.org/wiki/Talk:TransLink_(South_East_Queensland)_services

When the limit is hit, there will be HTML comments in the parser output 
indicating the problem. For all articles which use more than 1KB of 
templates, there will be an HTML comment at the bottom of the parser output 
showing how close the article is to the size limit. This allows technically 
capable users to monitor and optimise the expansion size of their templates.

At this stage I would like to invite comments. The we can move forward to a 
trial, perhaps in a few days, if there are no objections.

-- Tim Starling