User:Cscott/Ideas/A Plan for Scribunto

Introduction: The State of Scribunto

Scribunto is the primary template system used in Wikimedia projects since its introduction in 2013. Scribunto was intended to curb runaway template complexity and separate the content and presentation layer.^[1] Although Scribunto was successful at improving template evaluation performance, its other goals are largely unrealized. Scribunto provides a secure sandbox for modules written in Lua 5.1, which has been end-of-life since February 2012.^[2] Scribunto has limited localization support, which has spawned ad-hoc solutions such as Module:TNT and the Global templates proposal on MediaWiki. Further, Scribunto doesn't support modern software development processes such as code review and fork/merge version control.

Step 1: Prepare for Change

Before we can build any new functionality into Scribunto, we need to address the fact that the Lua 5.1 sandbox is unsustainable. In addition to security concerns, the obsolete Lua version creates barriers to entry for editors since modern textbooks, university courses, or educational materials don't document this version of Lua. We need to begin to move to a supported version of Lua and create the infrastructure required to enable further upgrades on a regular schedule. Fortunately, the Parsoid project has created much of the needed infrastructure: it uses a canary server running "the next version" and tools ("round trip" and "visual diff" testing, in the case of Parsoid) to compare the rendering of tens of thousands of project pages between the "previous" and "next" versions of the software-under-test. Production is only transitioned to the "next" version once confidence is reached that the impact on our project content is acceptable.^[3]

Although it is technically possible in many cases to rewrite Lua code such that it functions identically on both Lua 5.1 and a newer Lua release, more conservatively we can assume that there will be some pages for which explicit versioning is required: either separate versions of the module for different Lua versions will be needed or we'll need to explicitly mark that a module is/is not compatible with a specific Lua version.

Step 2: Allow for multiple engines

Scribunto calls the component responsible for executing module code an "engine", and already supports multiple engines—just not simultaneously. Scribunto ships with support for running code both in an embedded sandbox (the luasandbox PHP extension) and by shelling out to a command-line Lua binary.

In order to pragmatically migrate code from Lua 5.1 to a newer Lua version, we will need to be able to run multiple Lua engines simultaneously: an editor will want to be able to test with the newer Lua version in production while we still use Lua 5.1 for most renders. One difficulty here is that manner in which luasandbox embeds Lua as a PHP extension does not support embedding multiple versions of Lua in the same PHP binary.^[5]

On the other hand, it is possible to cross-compile newer versions of Lua to run in a Lua 5.1 engine, using a compiler written in Lua and run under Scribunto itself. The page User:Cscott/Ideas/Improved for-loops for Lua constitutes a live demo; although it doesn't loudly advertise the fact that its source language is Lua 5.4 as opposed to Lua 5.1, the first few lines of the source code document this fact. Similarly, it ought to be possible to cross-compile Lua 5.1 to execute under a newer Lua runtime. Cross-compilation may not be 100% accurate, but it may provide additional compatibility aids that can be combined with the canary approach described above to identify and correct edge cases where cross-compilation is not 100% faithful. This could limit the number of pages which need explicit porting.^[6]

Step 3: Separate code and presentation, fragment data type improvements

Although separation of code and presentation was part of the original design goals for Scribunto, current Scribunto modules still contain a large amount of wikitext and other presentation layer strings and markup embedded into the module. This impairs localization, since localizable strings are embedded inside code, and also raises a barrier to entry since editors who may wish only to adjust presentation details are faced with the task of understanding the Scribunto logic layer in order to do so.

It is not strictly necessary to address this deficiency in order to make the Lua sandbox sustainable or provide "global templates", but because many templates will likely be at least partially rewritten in order to port them to Lua 5.5 or (as future steps provide) improve localization or development process, I recommend addressing the structural issues with the Scribunto API at this step so that rewrites can use the cleaner mechanics.

Step 4: Localization

There are four aspects of Scribunto modules which require localization: the module name, the module Lua code (embedded function and variable names, comments, etc), the named arguments to the {{#invoke}} of the Scribunto module, and any embedded wikitext or other content in the code or presentation layer. There have been a number of prior proposals to address template localization, including Module:TNT and the Global templates proposal on MediaWiki.

We will handle localization of the module name in the next step. Localization of the Lua code itself can be addressed with the cross-compiler mechanism from step 2, as demonstrated in the French example from this live demo.^[10]

Step 5: Version Control, Code Review, and Distribution

Scribunto does not support modern software development processes. The simple "file-based" version control system of MediaWiki's article space was superceded for software when CVS replaced RCS in the 1990s. Module code written on-wiki lacks fork/merge support, code review, and continuous integration testing. Further, certain common templates and modules are replicated on hundreds of different wiki projects without any built-in mechanism to keep them in sync.

Attempts to address this include proposals to create a specific repository for global templates^[15] and tools like Synchronizer. The Synchronizer tool in particular already provides a solid basis for synchronization of templates and modules across wikis, embodying a workflow favored by editors which gives explicit manual control of the synchronization process. This prevents an update to one copy from invisibly breaking uses on hundreds of other wikis; instead local admins are empowered to control the distribution of updates on their own wikis. Wikidata is used to map template names between wikis, in the same manner used by interlanguage links.^[16]

Step 6: Other improvements

To reduce barriers to entry for new editors, the Scribunto API should be gradually unified with the other platform APIs in JavaScript and PHP, so that method names and class structures are substantially the same across platform. This may entail the creation of a formal IDL mapping spec (such as the PHP mapping for WebIDL in IDLeDOM) so that method, class, and argument names are semi-automatically kept in sync across platforms and languages.

Scribunto has an HTML-creation library; it should probably be replaced by a DOM and SVG library to facilitate the creation of structured content.

Summary and Conclusion

This proposal outlines a plan for Scribunto which addresses its current maintainability crisis, more fully fulfills its original vision of code/presentation separation, allows full localization of templates, and modernizes the code development process used in creating and maintaining Scribunto modules.

Scribunto would become the second major feature using the Multi-Content Revision system, helping actualize the potential of MCR and pave the way for further feature use. Slots would be added for code content, a script version index with provisions for cross-compilation, a message bundle, a wikitext presentation layer, a git commit hash for the last synchronization point, and, as additional optional features, documentation, tests, and template styles. Existing modules would be migrated to a supported, maintainable, and editor-friendly Lua version using the same proven and repeatable process as used for the migration to Parsoid. Localization and development processes would similarly build on the proven processes used by translatewiki and gerrit/gitlab/github, respectively.

In addition, the necessary infrastructure would be present for more radical updates in the future: expanding the scope of VisualEditor to support many template use cases; supporting JavaScript as either a Scribunto module language, a Scribunto engine runtime, or both; fully localizing Lua at the comment and API level for a true multilingual code environment; further integration with Continuous Integration and other modern code development tools; and turning existing "no code" templates into Scribunto modules to apply the localization and other benefits of this proposal and eventually reduce the number of distinct template mechanisms in core.

Appendix: Is this "Global Templates"?

Using the taxonomy defined by Some thoughts on Global Templates, so called, let's evaluate this proposal. As described in "before you begin" at the head of this page, this proposal focuses on templates defined by Scribunto, and adds a mechanism to port "regular wikitext" transclusions and "parser functions" transclusions to Scribunto in step 3. This covers a large number of the template use cases. This proposal doesn't directly address style consistency, page layout, semantic annotations, or work flow, although synchronized Scribunto modules could be used for these, as they currently are.^[19] From the translator perspective, the localization mappings of module name (step 5) and module arguments (step 4) facilitate translating content which uses modules from one wiki to another.

Moving on to "global", there are three aspects defined in the taxonomy:

• On a database/infrastructure level, we do not include a template directly from a different wiki project; instead we rely on a semi-automated synchronization mechanism which requires explicit approval by administrators. This preserves local autonomy and does not require additional cross-wiki dependency mechanisms to be built. It could be further automated in the future in selected cases, for example to automatically approve changes to a certain set of templates once a week for a particular wiki.
• We do not build a global community of editors. Coordination of changes happens on gerrit/gitlab/github (which have limited social features) or on existing "home" wikis for template development. Editors may develop a convention to identify where discussion of changes for a particular template should occur, which could be as simple as a link in the template documentation.
• We do provide for localizable language-independent modules, as described in step 4.

The proposal does not require a standalone "template" wiki,^[15] although a standalone wiki could be created, for example to facilitate community-building. Although this would likely move primary development of modules to the "template" wiki, we expect that distribution of modules to other wikis would still be done via git to ensure control by local admins of the update timing. We don't recommend building specific code-development tools into a standalone wiki; those already exist in the various open source code forges and we should reuse that work rather than reinvent it.

The taxonomy also described "types". We expect input and output types for Scribunto modules to be defined by TemplateData (T395968, T55413) which is the third "type" defined in the taxonomy, although at a technical level providing for PFragment input arguments and output (step 3) facilitates passing structured data. In particular, it is expected that the Wikifunctions ZObject will eventually become a PFragment subtype, which will provide access to other types in the taxonomy. Other types and semantic information (for example a Wikidata item selector, or Wikidata-based constraints or annotations on values) can be incorporated into TemplateData, and are not directly part of the module/template system.

Finally, the "theory of change" embodied by this proposal is moderate/incrementalist. Each step in this proposal is intended as a modest change to an existing product (Scribunto) which doesn't immediately break existing uses. However, community participation is required to adopt the new framework. In the first steps, we need community support to transition modules from Lua 5.1 to Lua 5.5, and we expect a mix of collaboration with community (especially around issues which can be identified by the Linter extension) and a modest amount of foundation resources devoted to porting key modules. This migration/linter model has been previously used with success in the HTML5 Tidy effort and the Parsoid Read Views project.

Further progress toward module localization requires community participation to port existing modules to the new framework and to establish git repositories for key translatable modules. Unfortunately the needs of minority-language Wikipedias, which are most benefited by translatable and reusable modules, are not often visible to editors on English and German Wikipedia, where much template/module development occurs. We have made an effort to make porting modules to the new framework low-friction, but one can expect evangelism and foundation resources (focused edit-a-thons, for example) will be needed to shift the window such that localizable modules are considered standard, not the exception. Once established and accepted by the community, we can apply the migration/linter model to fully convert existing modules and ultimately deprecate other templating mechanisms. Ultimately all newly-created modules will come pre-equipped with translation message bundles, a wikitext slot, and documentation marked for translation, and requesting a gerrit or gitlab repository for the module is a matter of a few clicks when the module is first desired for use on a different wiki.

References