Status: Reviewed, not yet implemeted
Author: Dmitry Lomov
There is a number of requests that require Skylark API to access functionality of native rules from Skylark rules. Typical scenarios can be illustrated by the following "sandwich":
bread_library(name = "top", …) java_library(name = "meat", deps = [":top", …], …) bread_library(name = "bottom", deps = [":meat", …])
Here bread_library is a rule written in Skylark. Here we need three things:
In this document we present some ideas about what how this all might look like, and suggest some practical steps we can take to get there.
This proposal assumes Declared Providers are implemented. Here is how the implementation of bread_library might look like:
# Implementation of a rule that transpiles to Java and invokes a native # compilation def _bread_library_impl(ctx): bread_sources = [f for src in ctx.attrs.src for f in src.files] generated_java_files = _invoke_bread_transpiler(ctx, bread_sources) # lang.java.provider is a declared provider for Java java_deps = [target[lang.java.provider] for target in ctx.attrs.deps] # create a native compilation action java_p = lang.java.compile(ctx, srcs = generated_java_files, # information about dependencies is just a lang.java.provider deps = java_deps, ...) # java_p is a lang.java.provider representing the result of compilation # action we return that provider and immediately java_libary rule can depend # on us return [java_p, ...] # Implementation of a rule that compiles to JVM bytecode directly def _scala_library_impl(ctx): # collect dependency jars to pass to the compile action dep_jars = [dep[java.lang.provider].jar for dep in ctx.attrs.deps] jar_output = ctx.new_file(...) ... construct compilation actions ... # build a provider that passes all transitive information transitive_p = lang.java.transitive( [dep[java.lang.provider] for dep in ctx.attrs.deps]) java_p = lang.java.provider( transitive_p, jar = jar_output, # update transitive information that we care about transitive_jars = transitive_p.transitive_jars | set(jar_output), ... whatever other information is needed ...) # return java.lang.provider return [java_p, ...]
The provider is the glue ("butter") that connects Skylark rules to native rules
and also to the native rule implementations exposed to Skylark. Note how the
native rule implementation (lang.java.compile) both consumes the entire
providers from dependencies and returns the provider that needs to be returned
from the rule.
lang.java.transitive is a function that passes all the
transitive information correctly from dependencies. The existing '.java'
becomes the same thing as lang.java.provider.
Note: for the sake for this document we are placing things in lang.java. There are other alternatives to this, e.g. "magical" .bzl files from which java_provider and java_compile function are exported.
Our current native rules are not as neat as described above. Making them extensible in one go is a difficult and long term project (or rather, projects: one for each language). Here is a suggested steps for extensibility of particular language implementation (we continue to use Java as a running example).
At this step, lang.java.provider is a black box. Skylark rules cannot construct the lang.java.provider directly: the only way to create it is to invoke lang.java.compile function.
Native implementations of Java rules are rewritten so that they can link to deps that return lang.java.provider and that they return lang.java.provider. The implementation of the provider can just be a bag of all providers that Java rules normally return - since that bag is not openable by Skylark, we can refactor it later without much difficulty .
Native compilation function (java.lang.compile) is pretty much JavaLibrary.create refactored so that it gets its dependent providers not from attributes but from a list of bags. JavaLibrary.create just collects the bags from deps and passes those to that function.
At the end of this phase, implementing code generators (and code generating aspects) such as java_proto_library becomes possible. This also covers many (most?) use cases where people use macros to delegate to native rule implementations
No huge refactoring of language rule implementation is needed, but the stage is set for gradual opening up in the future.
(Optional) As lang.java.provider is just a bag of existing providers, it is easy to just implement everything in 'target.java' on top of it, if desired.
The next step in the API evolution is making the black box provider less black. This means introducing a constructor for lang.java.provider as well as accessors to fields.
The API can be designed gradually and thoughtfully, only exposing the things we need and adding carefully: as an example sequence first just java libraries, then resources, then JNI, then support for tests. Existence of lang.java.transitive is crucial at this stage as it allows merging of transitive information from dependencies that is not yet exposed to Skylark.
As API exposure gradually progresses, the exposed Skylark API reaches parity with internal API.
Through the execution of this phase, more and more use cases are covered, and at the end the rules are fully extensible.