Notes on the design of the Melbourne Mercury compiler

• mercury_compile_llds_backend.m for the LLDS backend, and
• mercury_compile_mlds_backend.m for the MLDS backend.

The LLDS backend is so named because its main data structure is the ``low-level data structure'', which is the compiler's representation of imperative code at a level that is only slightly above assembly code (pseudo)instructions, containing references to virtual machine registers (that may correspond to real machine registers), stack slots, labels, and jumps. The LLDS backend was the first backend, and it generates low-level C, preferably with GNU C extensions.

The MLDS backend is so named because its main data structure is the ``medium-level data structure'', which is the compiler's representation of imperative code at a level that corresponds to handwritten C or Java code, containing e.g. assignment, if-then-else, switch and while statements inside functions and/or class methods. The MLDS backend was the second backend, and can generate (standard) C, Java or C#.

The modules in the compiler are structured by being grouped into "packages". A "package" is just a meta-module, i.e. a module that contains other modules as submodules. Every submodule is stored in a separate file. (In most cases, the name of this file is just the final component in their full module name, though in a few cases, it includes some of the earlier components as well.) We have a package for the top-level, a package for each main pass, and finally there are also some packages for library modules that are used by more than one pass.

Taking all this into account, the package structure looks like this:

• At the top of the dependency graph is the top_level.m package, which currently contains only the mercury_compile*.m modules, which invoke all the different ``ends'' in the compiler. (The name ``middle-end'' sounds funny, but it fits in with ``front-end'' and ``back-end'', both of which are traditional names in compiler design.)
• The next level down are all of these ``ends'', each of which consists of several passes. In general, we try to stick by the principle that later passes can depend on data structures defined in earlier passes, but not vice versa.
  • the front-end
    • 1. parsing (source files -> HLDS)
      Packages: parse_tree.m and hlds.m
    • 2. semantic analysis and error checking (HLDS -> annotated HLDS)
      Package: check_hlds.m
  • the middle-end
    • 3. high-level transformations (annotated HLDS -> annotated HLDS)
      Packages: transform_hlds.m and analysis.m
  • the back-ends
    • a. LLDS back-end
      Package: ll_backend.m
      • 3a. LLDS-back-end-specific HLDS->HLDS transformations
      • 4a. code generation (annotated HLDS -> LLDS)
      • 5a. LLDS level optimizations (LLDS -> LLDS)
      • 6a. output code (LLDS -> C)
    • b. MLDS back-end
      Package: ml_backend.m
      • 3a. MLDS-back-end-specific HLDS->HLDS transformations
      • 4b. code generation (annotated HLDS -> MLDS)
      • 5b. MLDS level optimizations (MLDS -> MLDS)
      • 6b. output code (MLDS -> C or MLDS -> C# or MLDS -> Java, etc.)
    • c. bytecode back-end
      Package: bytecode_backend.m
      • 4c. code generation (annotated HLDS -> bytecode)
    • There is also the backend_libs.m package which contains modules which are shared between several different back-ends.
• Finally, at the bottom of the dependency graph there is the package libs.m, which contains the option handling code, and many utility modules that are not sufficiently general or sufficiently useful to go in the Mercury standard library.

In addition to the packages mentioned above, there are also packages for the build system:

• make.m contains the support for the `--make' option, and
• recompilation.m contains the support for the `--smart-recompilation' option.

Option handling

The command-line options are defined in the module options.m. mercury_compile.m calls library/getopt.m, passing the predicates defined in options.m as arguments, to parse them. It then invokes handle_options.m (and indirectly, op_mode.m and compute_grade.m) to postprocess the option set. The results are represented using the type globals, defined in globals.m. (Part of the globals structure is defined in optimization_options.m, which is generated automatically from a higher-level specification.) The globals structure is available in the HLDS representation, but it is passed around as a separate argument both before the HLDS is built and after it is no longer needed.

After we have processed the options, the rest of the action is co-ordinated either from mercury_compile.m or from make.m, depending on whether the `--make' option is in effect.

The rest of this document contains a brief introduction each module of the compiler. If you want more information about either the purpose or the design of a particular module, see the documentation at the start of that module's source code.

Build system

Support for `--make' is in the make.m package, which contains the following modules:

• make.top_level.m categorizes targets passed on the command line and passes them to the appropriate module to be built.
• make.track_flags.m keeps a database that records which options were used to compile each module.
• make.program_target.m handles whole program `mmc --make' targets, including executables, libraries and cleanup.
• make.module_target.m handles targets built by a compilation action associated with a single module, for example making interface files.
• make.dependencies.m computes what other targets a target depends on.
• make.check_up_to_date.m does what its name says: it checks whether the files that a target depends on are up-to-date.
• make.deps_map.m provides an efficient representation of purpose-specific sets for use by make.dependencies.m.
• make.deps_cache.m provides caches for make.dependencies.m.
• make.find_local_modules.m finds the modules in the current directory that are reachable from a given module.
• make.module_dep_file.m records dependency information for each module between compilations in .module_dep files.
• make.get_module_dep_info.m uses the facilities exported by make.module_dep_file.m to read .module_dep files, or, if needed, to create them.
• make.build.m provides the machinery required to bring targets up to date.
• make.make_info.m defines the main data structure used by all the make*.m modules above.
• make.file_names.m provides mechanisms to compute the file names of make targets.
• make.hash.m contains code to hash the specifications of various entities that `mmc --make' deals with in order to make them smaller, which makes their comparisons more efficient.
• make.timestamp.m contains predicates to get files' timestamps, and to compare timestamps.
• make.util.m contains utility predicates.
• options_file.m reads the options files specified by the `--options-file' option. Also used by mercury_compile.m to collect the value of DEFAULT_MCFLAGS, which contains the auto-configured flags passed to the compiler.

Front end

1. Parsing

The parse_tree.m package

We transform the results of lexical analysis (done by library/mercury_term_lexer.m) into the Mercury parse tree in two stages:

• stage 1 parsing - converting token sequences to terms.

  library/mercury_term_parser.m contains the code to do this, while library/term.m and library/varset.m define the term and varset data structures, and predicates for manipulating them.

• stage 2 parsing - converting terms to `items' (declarations, clauses, etc.)

  The result of this stage is a parse tree that has a close correspondence with the source code.

(There is also a parsing stage 3, which transforms the parse tree into the HLDS, but that is not part of the parse_tree.m package itself.)

The parse tree data structure definition is in prog_data.m, prog_data_event.m, prog_data_foreign.m, prog_data_pragma.m, prog_data_used_modules.m, prog_item.m and file_kind.m, while the code to create it is in the following modules.

• find_module.m locates source files containing Mercury modules.
• parse_module.m handles the top level tasks of reading in whole Mercury source files, interface files, and optimization files.
• parse_item.m parses in the top level parts of items, particularly declarations.
• parse_goal.m parses goals.
• parse_dcg_goal.m parses goals (and clauses) using Definite Clause Grammar notation.
• parse_vars.m parses lists of variables.
• parse_type_name.m parses type names, while parse_inst_mode_name.m parses inst and mode names.
• parse_type_defn.m parses type definitions, while parse_inst_mode_defn.m parses inst and mode definitions.
• parse_type_repn.m parses items that the compiler automatically puts into interface files to give modules that import a type information about the representation of that type.
• parse_class.m parses typeclass and instance declarations, as well as typeclass and inst constraints on other declaration (such as predicate declarations).
• parse_pragma.m parses pragma declarations. It subcontracts some of its work to other modules:
  • parse_pragma_analysis.m parses pragmas that record analysis results;
  • parse_pragma_foreign.m parses pragmas that deal with foreign languages; and
  • parse_pragma_tabling.m parses tabling pragmas.
• parse_mutable.m parses initialize, finalize and mutable declarations.
• parse_sym_name.m parses symbol names.
• parse_error.m defines the types that represents the possible outcomes of parsing a source file.
• parse_util.m and parse_types.m define some types and predicates needed by the other modules above.

There are several modules whose collective job it is to print (parts of) the parse tree.

• The top levels of parse trees are output by parse_tree_out.m.
• parse_tree_out_item.m outputs most kinds of items.
• parse_tree_out_clause.m outputs clauses and goals.
• parse_tree_out_pragma.m outputs pragmas.
• parse_tree_out_pred_decl.m outputs (parts of) predicate, function and mode declarations.
• parse_tree_out_sym_name.m outputs sym_names.
• parse_tree_out_cons_id.m outputs cons_id.
• parse_tree_out_type.m outputs types.
• parse_tree_out_inst.m outputs insts and modes.
• parse_tree_out_term.m outputs variables and terms.
• parse_tree_out_misc.m outputs the lowest level, and smallest, parts of the parse tree.
• parse_tree_out_info.m defines the structure that stores the options used by all of the above modules.
• parse_tree_output defines the pt_output typeclass and its instances, also for use by all of the above modules.

There are several modules that provide utility predicates that operate on (parts of) the parse tree.

• builtin_lib_types.m contains definitions about types, type constructors and function symbols that the Mercury implementation needs to know about.
• pred_name.m contains code that constructs predicate names. As we create transformed versions of predicates and functions, this module constructs new names for them, derived from the name of the original version.
• prog_item_stats.m has facilities for gathering and printing statistics about the parse tree.
• prog_util.m contains some utility predicates for manipulating the parse tree.
• prog_detism.m contains utility predicates for manipulating determinism and determinism components.
• prog_mode.m contains utility predicates for manipulating insts and modes.
• prog_type.m contains utility predicates for manipulating types.
• prog_type_construct.m contains predicates for constructing types.
• prog_type_repn.m contains predicates for testing types' representations.
• prog_type_scan.m contains predicates for gathering the variables inside types.
• prog_type_subst.m contains predicates for performing type substitutions.
• prog_type_test.m contains predicates for performing simple tests on types.
• prog_type_unify.m contains predicates for checking whether two types unify and whether one subsumes the other.
• prog_rename.m contains predicates for performing variable substitutions.
• prog_foreign.m contains utility predicates for manipulating foreign code.
• prog_mutable.m contains utility predicates for manipulating mutable variables.
• prog_event.m contains utility predicates for working with events.
• error_spec.m defines the error_spec type, which is a structured representation of diagnostic messages.
• maybe_error.m contains types that allow the representation of the results of computations that either succeed, or generate some error_specs.
• write_error_spec.m contains code for generating nicely formatted printouts of error_specs.
• error_type_util.m, error_util.m, and error_sort.m contain utility predicates and functions operating on error_specs.

The second main use of the parse tree, besides transformation into HLDS, is the generation and consumption of interface files. This is done by the following modules.

• grab_modules.m figures out what interface files to read, and reads them. It also does a bunch of other semi-related things.
• check_import_accessibility.m checks whether imported (and used) modules are actually accessible (by having imports of all their ancestor modules), and generates error messages if the answer is "no".
• read_module.m has code to read in source, interface and optimization files.
• split_parse_tree_src.m splits up the parse tree of a source file (parse_tree_src) into a sequence of parse_tree_module_srcs, one per module contained in the source file.
• write_module_interface_files.m has the code to write out `.int0', `.int', `.int2', and `.int3' files.
• comp_unit_interface.m figures out which parts of a raw parse_tree_module_src belong in its .int3, .int0, .int and .int2 files.
• check_type_inst_mode_defns.m checks whether the items in a module related to a given type, inst or more are all consistent with one another. It uses prog_foreign_enum.m to check the validity of foreign enum declarations, using code that also does the same job for foreign export enum declarations.
• convert_parse_tree.m does conversions in both directions between the generic interface file parse tree on the one hand and the parse tree kinds specific to each kind of interface file on the other hand. The specific parse trees enforce structural invariants that the generic parse tree does not.
• decide_type_repn.m generates type_representation items that we put into interface files.
• canonicalize_interface.m has code to put the contents of `.int0', `.int', `.int2', and `.int3' files into a standard order, to prevent the need to recompile the modules that import them if the only change to a module is a reordering of some declarations (which causes no change in the module's semantics).
• check_module_interface.m checks whether a module exports anything.
• generate_dep_d_files.m generates the information from which write_deps_file.m writes out Makefile fragments.
• module_baggage.m contains the module_baggage type, the types of its components, and utility predicates.
• get_dependencies.m contains predicates that compute various sorts of direct dependencies (those caused by imports) between modules.
• module_dep_info.m defines data structures for recording, for a given module, which other modules it depends on, and how.
• deps_map.m and module_deps_graph.m contain data structures for recording indirect dependencies between modules, and the predicates for creating and using them.
• file_names.m does conversions between module names and file names. It uses java_names.m, which contains predicates for dealing with names of things in Java.
• source_file_map.m contains code to read, write and search the mapping between module names and file names.
• module_cmds.m handles the commands for manipulating interface files of various kinds.
• item_util.m contains some utility predicates dealing with items.

We do most, though not all, forms of module qualification on programs when they are represented as parse trees. This means that we add module qualifiers to all types, insts and modes, check that every referred-to type, inst and mode actually exists, and that there is only possible match. This is done in the parse_tree package because it must be done before the `.int' and `.int2' interface files are written. This code also checks whether imports are really needed in the interface.

This work is done by the modules of the module_qual.m package.

• module_qual.collect_mq_info.m collects information about what types, insts etc are defined in which modules.
• module_qual.qualify_items.m uses the collected information to module qualify items and their components.
• module_qual.id_set.m defines the data structure used by collect_mq_info and qualify_items to do their job and communicate with each other.
• module_qual.qual_errors.m handles the errors that arise when an item refers to an entity (type, or inst, or ...) that is either not defined anywhere, or is defined in more than one module, and the reference does not indicate which one is intended.

As to what is module qualified when:

• All types, typeclasses, insts and modes occurring in pred, func, type, typeclass and mode declarations are module qualified by module_qual.m and its submodules.
• All types, insts and modes occurring in lambda expressions, explicit type qualifications, and clause mode annotations are module qualified in make_hlds.m.
• Constructors occurring in predicate and function mode declarations are module qualified during type checking.
• Predicate and function calls and constructors within goals are module qualified during mode analysis.

The parse tree module also contains equiv_type.m, which does expansion of equivalence types, and of `with_type` and `with_inst` annotations on predicate and function type and mode declarations.

Expansion of equivalence types is really part of type-checking, but is done on the parse tree rather than on the HLDS because it turned out to be much easier to implement that way. (Though later we had to add a pass the expand equivalence types in the HLDS anyway.)

The hlds.m package

Once the stages listed above are complete, we then convert from the parse_tree data structure to another data structure which no longer attempts to maintain a one-to-one correspondence with the source code. This data structure is called the High Level Data Structure (HLDS), and it is defined in the hlds.m package.

The last stage of parsing is this conversion to HLDS, which is done mostly by the following submodules of the make_hlds subpackage in the hlds package.

• make_hlds_types.m defines the types used by many of the submodules of make_hlds.m.
• make_hlds_passes.m calls the other modules to perform the conversion.
• make_hlds_separate_items.m separates out the different kinds of items, so that when make_hlds_passes.m adds one kind of item (e.g. clauses) to the HLDS, it can rely on the fact that all items of another kind (e.g. predicate declarations) have already been processed.
• superhomogeneous.m performs the conversion of unifications into superhomogeneous form.
• state_var.m expands away state variable syntax.
• field_access.m expands away field access syntax.
• goal_expr_to_goal.m converts clauses from parse_tree format to hlds format. It also implements universal quantification (via the transformation `all [Vs] G' ===> `not (some [Vs] (not G))') and implication (using `A => B' ===> `not(A, not B)').
• add_clause.m oversees the conversion of clauses from parse_tree format to hlds format. It handles their addition to procedures, which is nontrivial in the presence of mode-specific clauses.
• add_pred.m handles type and mode declarations for predicates. (Actually this is in the hlds package because it is called by add_special_pred.m, but does most of its work for the other modules in the make_hlds package.)
• default_func_mode.m: if a function has no declared mode, this module adds to it the standard default mode.
• add_type.m handles the declarations of types.
• add_mode.m handles the declarations of insts and modes, including checking for circular insts and modes.
• add_solver.m adds to the HLDS the casting predicates needed by solver types.
• add_mutable_aux_preds.m adds to the HLDS the auxiliary predicates (init, get, set, lock, unlock) needed by mutables.
• add_class.m checks and records typeclass and instance declarations.
• instance_method_clauses.m generates the definitions of the predicates that implement instance methods.
• qual_info.m handles the abstract data types used for module qualification.
• make_hlds_warn.m looks for constructs that merit warnings, such as singleton variables and variables with overlapping scopes.
• make_hlds_error.m defines any error messages that are used by more than one submodule of make_hlds.m. (Actually, make_hlds_error.m is in the hlds package, because it is needed by some other modules that have been moved from the make_hlds package to hlds.)
• add_foreign_proc.m adds foreign procs (Mercury predicates defined using foreign language code) to the HLDS.
• add_foreign_enum.m adds foreign enums (Mercury enum types linked to foreign language equivalents) to the HLDS.
• add_pragma_tabling.m adds everything needed to implement tabling pragmas to the HLDS.
• add_pragma_type_spec.m adds everything needed to implement type specialization pragmas to the HLDS.
• add_pragma.m adds the easiest-to-implement kinds of pragmas to the HLDS, i.e. those that don't need their own module.

The HLDS data structure itself is spread over the following modules:

• hlds_args.m defines the parts of the HLDS concerned with predicate and function argument lists.
• hlds_data.m defines the parts of the HLDS concerned with data types, and the representation of values of various types;
• hlds_cons.m defines the parts of the HLDS concerned with function symbols.
• hlds_inst_mode.m defines the parts of the HLDS concerned with instantiation states and modes.
• hlds_class.m defines the parts of the HLDS concerned with type classes and type class constraints.
• hlds_goal.m defines the part of the HLDS concerned with the structure of goals, including the annotations on goals.
• hlds_clauses.m defines the part of the HLDS concerning clauses.
• hlds_rtti.m defines the part of the HLDS concerning RTTI.
• const_struct.m defines the part of the HLDS concerning constant structures.
• hlds_pred.m defines the part of the HLDS concerning predicates and procedures;
• pred_table.m defines the tables that index predicates and functions on various combinations of (qualified and unqualified) names and arity.
• hlds_promise.m defines the parts of the HLDS concerned with recording promises made by the programmer about the properties of predicates and functions.
• hlds_module.m defines the top-level parts of the HLDS, including the type module_info.
• status.m defines the type that record the import/export status of entities such as types, insts, modes, and predicates.
• vartypes.m defines the data structure that maps each variable to its type. New code should use var_table.m in all passes after typechecking.
• var_table.m defines the data structure that maps each variable to its name, type, and an indication of whether that type is a dummy type. var_table.m itself is part of the parse_tree module (because some of its uses occur before the construction of the HLDS), so the operations on var_tables that need access to the HLDS are in var_table_hlds.m.
• Some modules need to provide functionality both to modules that use var_tables and other modules that use varsets, vartypes, or both. var_db.m defines operations that can provide information about variables' names and/or types that can operate on top of either var_tables, or its alternatives.

• du_type_layout.m decides how values of discriminated union types are laid out in memory. The two main issues it handles are floats (which can be a problem on platforms where they don't fit in words) and the packing of data structures as densely as possible.

  Once the representations of all types are known, this module invokes add_special_pred.m to declare, and if need be, define the unify and compare predicate of each type.

• add_special_pred.m adds unify, compare, and (if the compare predicate needs it) index predicates to the HLDS.

The submodules of the hlds_out.m package contain predicates to dump the HLDS to a file. These predicates print all the information the compiler has about each part of the HLDS. The module hlds_desc.m, by contrast contains predicates that describe some parts of the HLDS (e.g. goals) with brief strings, suitable for use in progress messages used for debugging.

The module hlds_defns.m contains code to print the set of definitions in the HLDS to a file. When dividing a module into two or more submodules, one can use the information thus generated to check whether the new modules include every type, inst, mode, predicate and function definition in the original module exactly once. (The other sorts of definitions, e.g. typeclass definitions, are typically so few in number that one can keep track of them in one's head.)

The module hlds_call_tree.m contains code to print out the call tree of the local predicates and functions in the HLDS, ordered by a depth-first left-to-right traversal. This can suggest to programmers an order that the predicates and functions of the module can be put into in the source file, or at least provide a starting point for such an order.

The hlds.m package also contains some utility modules that contain various library routines which are used by other modules that manipulate the HLDS:

• hlds_proc_util.m contains predicates that operate on procedures.
• mark_tail_calls.m marks directly tail recursive calls as such, and marks procedures containing directly tail recursive calls as such.
• hlds_code_util.m contains utility routines for use during HLDS generation.
• goal_form.m contains predicates for determining whether HLDS goals match various criteria.
• goal_util.m contains various miscellaneous utility predicates for manipulating HLDS goals, such as attaching features to goals.
• make_goal.m contains predicates for creating new HLDS goals.
• passes_aux.m contains code to write progress messages, and higher-order code to traverse all the predicates defined in the current module and do something with each one.
• introduced_call_table.m contains what is effectively a database of the predicates and functions that may have calls introduced to them by various compiler passes. The reason why we need this is that if dead_proc_elim.m deleted those predicates and functions from the HLDS as dead code before the pass that introduces calls to them has been run, those calls would have no destination. By consulting this database, dead_proc_elim.m can avoid this problem.
• hlds_error_util.m contains utility routines for printing nicely formatted error messages for symptoms involving HLDS data structures. For symptoms involving only structures defined in prog_data, use parse_tree.error_util.
• error_msg_inst.m contains utility routines for printing insts and modes in nicely formatted error messages.
• code_model.m defines a type for classifying determinisms in ways useful to the various backends, and utility predicates on that type.
• arg_info.m contains utility routines that the various backends use to analyze procedures' argument lists and decide on parameter passing conventions.
• hhf.m contains facilities for translating the bodies of predicates to hyperhomogeneous form, for constraint based mode analysis.
• inst_graph.m defines the inst_graph data type, which describes the structures of insts for constraint based mode analysis, as well as predicates operating on that type.
• from_ground_term_util.m contains types and predicates for operating on from_ground_term scopes and their contents.

2. Semantic analysis and error checking

This is the check_hlds.m package, with support from the mode_robdd.m package for constraint based mode analysis.

Any pass which can report errors or warnings must be part of this stage, so that the compiler does the right thing for options such as `--halt-at-warn' (which turns warnings into errors) and `--error-check-only' (which makes the compiler exit at the end of semantic analysis).

• quantification.m handles implicit quantification and computes the set of non-local variables for each sub-goal. It also expands away bi-implications (unlike the expansion of implication and universal quantification, this expansion cannot be done until after quantification). This module is part of the hlds.m package rather than the check_hlds.m package, partly because it is rerun by several passes after semantic analysis to update nonlocal sets after changes to procedure bodies. The first invocation of quantification may be preceded by a pre-quantification pass (in pre_quantification.m), which can insert implicit existential quantifiers into trace goals to implement a scope rule about such goals that people tend to intuitively expect.
• check_typeclass.m both checks that instance declarations satisfy all the appropriate superclass constraints (including functional dependencies) and performs a source-to-source transformation on the methods from the instance declarations. The transformed code is checked for type, mode, uniqueness, purity and determinism correctness by the later passes, which has the effect of checking the correctness of the instance methods themselves (i.e. that the instance methods match those expected by the typeclass declaration). During the transformation, pred_ids and proc_ids are assigned to the methods for each instance.

  While checking that the superclasses of a class are satisfied by the instance declaration, a set of constraint_proofs are built up for the superclass constraints. These are used by polymorphism.m when generating the base_typeclass_info for the instance.

  This module also checks that there are no ambiguous pred/func declarations (that is, it checks that all type variables in constraints are determined by type variables in arguments), checks that there are no cycles in the typeclass hierarchy, and checks that each abstract instance has a corresponding typeclass instance.

• inst_check.m checks that all user defined bound insts are consistent with at least one type in scope (i.e. that the set of function symbols in the bound list for the inst are a subset of the allowed function symbols for at least one type in scope). The compiler generates a warning if it finds any user defined bound inst that is not consistent with any of the types in scope.
• inst_user.m performs on user defined bound insts the tests whose results the compiler needs, and records the results in the insts themselves. This is faster than having the compiler perform those tests repeatedly each time it needs the results of those tests.
• headvar_names.m tries to replace compiler-generated names of the form HeadVar__n with actual names given by the programmer.

  For efficiency, this phase not a standalone pass, but is instead invoked by pre_typecheck.m.

• The type checker consists of several modules.
  • pre_typecheck.m does some chores that need to be done before typechecking and which cannot be done earlier.
  • typecheck.m handles the top level management of typechecking, including managing the type inference process, and handling stubs.
  • typecheck.m handles the type checking of predicate and function definitions, including overloading resolution and module name resolution, and almost fully qualifies all predicate and functor names. It sets the map(var, type) field in the pred_info. However, it doesn't figure out the pred_id for function calls or calls to overloaded predicates. That can't be done in a single pass of typechecking, and so it is done later on (in purity.m for overloaded predicate calls, and in resolve_unify_functor.m for function calls).
  • type_assign.m and typecheck_info.m define the main data structures used by typechecking.
  • typecheck_error_overload.m, typecheck_error_undef.m and typecheck_errors.m generate error messages for type errors, with support from typecheck_error_type_assign.m and typecheck_error_util.m.
  • typecheck_msgs.m generates other kinds of messages from the type checker, such as those containing inferred type declarations, and diagnostics for non-contiguous clauses.
  • typecheck_debug.m helps debug the typechecker itself.
  • typeclasses.m checks typeclass constraints, and any redundant constraints that are eliminated are recorded (as constraint_proofs) in the pred_info for future reference.
  • type_util.m contains utility predicates dealing with types that are used in a variety of different places within the compiler.
  • post_typecheck.m may also be considered to logically be a part of typechecking, although it also prepares for mode analysis. It contains tests for errors such as unbound type and inst variables, unsatisfied type class constraints, and indistinguishable predicate or function modes. These tests can't be done in the main type checking pass, because they depend on type analysis being already complete.
  • types_into_modes.m propagates type information into modes. It is invoked as part of the post_typecheck pass, since it is also a task that must be done before mode analysis while needing checked type information.
  • check_for_missing_type_defns.m checks for locally defined types that have an abstract definition but no corresponding concrete definition.
• old_type_constraints.m contains an old (2008-2009) attempt by a summer student at a constraint based type analysis algorithm, which covers only a subset of Mercury.
• type_constraints.m contains a (start on) another constraint based type analysis algorithm, which is intended to cover all of Mercury.
• assertion.m (in the hlds.m package) is the abstract interface to the assertion table. Currently all the compiler does is type check the assertions and record for each predicate that is used in an assertion, which assertion it is used in. The set up of the assertion table occurs in post_typecheck.finish_assertion.
• purity.m is responsible for purity checking, as well as defining the purity type and a few public operations on it. It also does some tasks that are logically part of typechecking but which cannot be done until after the main part typechecking is complete. (This is separate from the work done by post_typecheck.m.) purity.m also does two other miscellaneous tasks. The first is the elimination of double negations; that needs to be done after quantification but before mode analysis. The other is converting calls to `private_builtin.unsafe_type_cast/2' into `generic_call(unsafe_cast, ...)' goals.
• check_promise.m records each promise in the appropriate table (the assertion table or the promise_ex table), and removes them from further processing as predicates.
• implementation_defined_literals.m replaces unifications of the form Var = $name by unifications to string or integer constants.
• polymorphism.m, with the help of polymorphism_clause.m and polymorphism_goal.m, handles the introduction of type_info arguments for polymorphic predicates and introduction of typeclass_info arguments for typeclass-constrained predicates. This phase needs to come before mode analysis so that mode analysis can properly reorder code involving existential types. (It also needs to come before simplification so that simplify.m's optimization of goals with no output variables doesn't do the wrong thing for goals whose only output is the type_info for an existentially quantified type parameter.)

  polymorphism_info.m defines the main data structures used by the polymorphism transformation, and provides basic operations on them. These are used only by polymorphism.m and polymorphism_type_info.m.

  polymorphism_type_info.m provides predicates to construct new type_infos and to extract existing type_infos from typeclass_infos. Most of these operations are used only by polymorphism.m, but some are also used by other modules.

  polymorphism_type_class_info.m provides predicates to construct new type_class_infos and to extract existing type_infos from typeclass_infos. These operations are used only by polymorphism.m.

  polymorphism.m subcontracts parts of its job to introduce_exists_casts.m, which is sometimes also invoked from modes.m.

  The polymorphism phase also converts higher-order predicate terms into lambda expressions. This is done in polymorphism_lambda.m, which is also invoked from modecheck_unify.m.

  When it has done most of its work, polymorphism.m calls clause_to_proc.m to make duplicate copies of the clauses for each different mode of a predicate; all later stages work on procedures, not predicates.

  A part of the work of polymorphism.m is done after copying clauses to procedures, though it is not clear whether this is by design or by accident. This part is done by polymorphism_post_copy.m.

• The mode analyser consists of several analysis modules and a whole bunch of service modules.

  These are the analysis modules.

  • modes.m is the top module of the mode analyser. It checks that procedures are mode-correct.
  • modecheck_goal.m does most of the work. It handles the tasks that are common to all kinds of goals, including annotating each goal with a delta-instmap that specifies the changes in instantiatedness of each variable over that goal, and does the analysis of several kinds of goals.
  • modecheck_conj.m is the submodule which analyses conjunctions It reorders code as necessary.
  • modecheck_unify.m is the submodule which analyses unifications. It also module qualifies data constructors.
  • modecheck_call.m is the submodule which analyses calls.

  These are the service modules.

  • mode_info.m The main data structure for mode analysis.
  • delay_info.m defines the most important component of mode_info, the data structure used for storing information for scheduling: which goals are currently delayed, what variables they are delayed on, etc.
  • modecheck_util.m contains utility predicates useful during mode analysis.
  • instmap.m defines the instmap and instmap_delta ADTs, which store information on what instantiations a set of variables may be bound to. (This is in the hlds.m package, because most modules in the middle- and back-ends need the info in those ADTs.)
  • inst_match.m contains the code for examining insts and checking whether they match.
  • inst_abstract_unify.m contains code for unifying insts.
  • inst_lookup.m contains code for looking up insts, possibly expanding references to user-defined insts in the process.
  • inst_merge.m contains code for merging insts. When the branches of an if-then-else or disjunction all bind a variable, this merging process figures out what the variable's inst should be when execution reaches the program point just after the branched control structure.
  • inst_test.m contains various kinds of tests on insts.
  • inst_util.m contains code for creating new insts and modifying old insts.
  • mode_comparison.m module compares different modes of a predicate.
  • mode_errors.m contains all the code to generate error messages for mode errors.
  • proc_requests.m contains the queue of procedures that mode analysis has discovered a need for, but which don't yet exist. This may be a mode other than (in,in) for the automatically generated unify predicate for a type, a mode other than (out,in,in) for the automatically generated compare predicate for a type, or a new mode for a user-defined predicate or function whose set of modes is being inferred.
  • inst_mode_type_prop.m propagates type information into insts and modes.
  • recompute_instmap_deltas.m does what its name says: it recomputes the instmap deltas of the component goals of a procedure body. This needs to be done after virtually all changes to that body, so this module is used by many of the later stages of the compiler.
  • mode_test.m contains various kinds of tests on modes.
  • mode_top_functor.m computes the mode of the top functor of a term from the mode of the whole term. This is of interest because it controls e.g. when a target language variable that holds the term becomes alive.
  • mode_util.m contains miscellaneous useful predicates dealing with modes, many of which transform between the various representations of modes (from_to_insts, mer_mode, unify_mode).
  • mode_debug.m contains debugging code for tracing the actions of the mode checker.
  • delay_partial_inst.m adds a post-processing pass on mode-correct procedures to avoid creating intermediate, partially instantiated data structures.
• The constraint based mode analyser was an experimental alternative to the usual mode analysis algorithm. It worked by building a system of boolean constraints about where (parts of) variables can be bound, and then solving those constraints using reduced ordered binary decision diagrams (robdds). It has been abandoned in favor of the propagation based constraint solver, for two main reasons. First, its performance was far too dependent on finding a good ordering of variables for the robdds, and we found no heuristics that could ensure such an ordering. And second, even with the best orderings, the performance left a lot to be desired.
  • mode_constraints.m finds the constraints and adds them to the constraint store.
  • mode_ordering.m uses solutions of the constraint system to find an ordering for the goals in conjunctions.
  • mode_constraint_robdd.m is the interface to the modules that perform constraint solving using reduced ordered binary decision diagrams (robdds).
  • We had several implementations of solvers using robdds. Each solver was in a module named mode_robdd.X.m, and they all belonged to the top-level mode_robdd.m. We have kept only one of these modules, mode_robdd.tfeirn.m.
• The new propagation based constraint based mode analyser is the proposed replacement for the constraint based mode analysis algorithm. It performs conjunct reordering for a subset of Mercury programs (it aborts if it encounters higher order code or a parallel conjunction, or is asked to infer modes).
  • prop_mode_constraints.m is the interface to the old mode_constraints.m. It builds constraints for an SCC.
  • build_mode_constraints.m is the module that traverses a predicate to build constraints for it.
  • abstract_mode_constraints.m describes data structures for the constraints themselves.
  • ordering_mode_constraints.m solves constraints to determine the producing and consuming goals for program variables, and performs conjunct reordering based on the result.
  • mcsolver.m contains the constraint solver used by ordering_mode_constraints.m.
  • goal_mode.m is intended to help implement the transition from the original mode analysis algorithm implemented by modes.m and related modules to the propagation based constraint solver. It is intended to represent an interface between mode analysis on the one hand, and the rest of the compiler on the other hand, that is sufficiently abstract that it could be implemented on top of both mode analysis systems. Initially, it will be implemented on top of the old mode analysis system. Once the rest of the compiler is transitioned to use this interface, we will transition its implementation to the propagation based constraint solver.

• The indexing and determinism analysis system also consists of several modules.
  • switch_detection.m transforms into switches those disjunctions in which several disjuncts test the same variable against different function symbols.
  • cse_detection.m looks for disjunctions in which each disjunct tests the same variable against the same function symbols, and hoists any such unifications out of the disjunction. If cse_detection.m modifies the code, it will re-run mode analysis and switch detection. This rerun of switch detection can then find switches that were buried too deeply for it to see in the original code cse_detection.m was invoked with.
  • det_analysis.m annotates each goal with its determinism; it inserts cuts in the form of "some" goals wherever the determinisms and delta instantiations of the goals involved make it necessary. Any errors found during determinism analysis are reported by det_report.m.
  • det_util.m contains utility predicates used in several modules.
• unique_modes.m checks that non-backtrackable unique modes are not used in a context which might require backtracking. Note that what unique_modes.m does is quite similar to what modes.m does, and unique_modes calls lots of predicates defined in modes.m to do it.
• The module stratify.m implements the `--warn-non-stratification' warning, which is an optional warning that checks for loops through negation. It was mainly used by the now-deleted Aditi backend.
• oisu_check.m checks whether the predicates mentioned in any pragmas about order independent state update obey the requirements placed on them by those pragmas.
• try_expand.m expands `try' goals into calls to predicates in the `exception' module instead.
• check_pragma_format_call.m checks the validity of any format_call pragmas recorded for the predicates and functions in the HLDS.
• Simplification finds and exploits opportunities for simplifying the internal form of the program, both to optimize the code and to massage the code into a form the code generator will accept. It also warns the programmer about any constructs that are so simple that they should not have been included in the program in the first place. (The fact that it can report warnings is why this pass needs to be part of the semantic analysis pass.)

  simplify.m is a package of submodules.

  • simplify_goal.m handles simplifications that involve the interaction of a goal with its environment, and then invokes one of the goal-type-specific submodules for further processing.
  • simplify_goal_call.m handles calls (plain, generic and foreign code). Using const_prop.m in the transform_hlds.m package, it attempts to partially evaluate calls to builtin procedures if the inputs are all constants.
  • simplify_goal_unify.m handles unifications. Amongst other things, it converts complicated unifications into procedure calls.
  • simplify_goal_conj.m handles conjunctions. It inlines nested conjunctions, eliminates unreachable code, and eliminates assign unification conjuncts (replacing the assigned-to variable with the assigned-from variable in the rest of the conjunction) if this is possible.
  • simplify_goal_disj.m handles disjunctions (and atomic goals). It eliminates unnecessary disjunction wrappers, and transforms semidet disjunctions into if-then-elses if this possible.
  • simplify_goal_ite.m handles if-then-elses (and negations). It warns about if-then-elses in which either the then-part or the else-part is unreachable. It also warns about if-then-elses that should be replaced by switches.
  • simplify_goal_switch.m handles switches. It eliminates switch arms that do nothing but fail, as well switch arms that cannot be reached (due to the arm being for function symbols that, according to its initial instantiation state, the variable being switched on cannot be bound to), It also eliminates switches that have no arms left.
  • simplify_goal_scope.m handles scope goals. It eliminates unnecessary nested scopes, replaces from_ground_term_construct scopes with a single assignment unifications referencing a constant structure in a constant structure database (to eliminate the need for any later passes to traverse the scope), and evaluates compile-time conditions on trace goals, eliminating either the compile-time condition wrapper on the trace goal (if the condition is true) or the trace goal scope altogether (if the condition is false).
  • common.m looks for (a) construction unifications that construct a term that is the same as one that already exists, or (b) repeated calls to a predicate with the same inputs. It replaces both with assignment unifications. It is invoked by the goal-type-specific submodules above.
  • format_call.m looks for calls to predicates such as string.format and io.format. It reports calls in which the types of the values to be printed disagree with the format string, and/or calls in which the agreement cannot be established. It also attempts to partially evaluate the correct calls, essentially interpreting the format string at compile time, not runtime.
  • split_switch_arms.m optimizes code in which one or more switches on a variable are nested inside one arm of a larger switch on the same variable. It does so by making all the distinctions between the cons_id of that outermost arm that any of the switches nested inside of it do. By splitting up that arm according to those distinctions, it effectively replaces several switches with just one switch.
  • simplify_proc.m handles the top-level processing of procedures and their bodies.
  • simplify_info.m defines the data structure that is threaded through the code of the submodules above, containing the information those submodules need.
  • simplify_tasks.m defines the type that identifies the tasks that the simplification package may be asked to do. Simplification can be invoked at several different points in the compilation process; different invocations need to perform different subsets of the tasks that simplification is capable of.
• unused_imports.m determines which imports of the module are not required for the module to compile. It also identifies which imports of a module can be moved from the interface to the implementation.
• style_checks.m generates warnings if a predicate or function declaration is not followed immediately by all the mode declarations of that predicate or function, and for module bodies in which either the exported or nonexported predicates and functions have one order for their declarations and a different order for their definitions.
• xml_documentation.m outputs a XML representation of all the declarations in the module. This XML representation is designed to be transformed via XSL into more human readable documentation.

Middle end

3. High-level transformations

This is the transform_hlds.m package.

• The first pass of this stage does tabling transformations (table_gen.m). This involves the insertion of several calls to tabling predicates defined in mercury_builtin.m and the addition of some scaffolding structure. Note that this pass can change the evaluation methods of some procedures to eval_table_io, so it should come before any passes that require definitive evaluation methods (e.g. inlining).
• The next pass of this stage converts lambda expressions into higher-order predicate terms that refer to freshly introduced separate predicates (lambda.m). This pass needs to come after unique_modes.m to ensure that the modes we give to the introduced predicates are correct. It also needs to come after table_gen.m because it needs the final evaluation method of each predicate, and after polymorphism.m, because polymorphism.m doesn't handle higher-order predicate constants.
• The next pass implements a program transformation that solves a rare problem, one that arises when a predicate fills in a partially instantiated term, and that term's data representation uses direct_arg tags. In such cases, the relevant head var(s) need not only to be passed from the caller to the callee (to tell them the head var's top function symbol) but also from the callee back to the caller (to tell them the newly-filled-in argument of that function symbol). Normally, the second part is not needed, because the caller can get the value of the function symbol's argument(s) from the heap, but in the case of function symbols that use the direct_arg tag, that is not the case.

  This pass has to come after both (a) the simplification pass at the end of semantic analysis, and (b) the lambda expansion pass, since it is these passes that record for it which arguments of which procedures have this problem.

• The next pass also simplifies the HLDS by expanding out the atomic goals that implement Software Transactional Memory (stm_expand.m).
• equiv_type_hlds.m expands type equivalences which are not meant to be visible to the user of imported modules. This is necessary for the Java and C# back-ends and in some cases for `:- pragma foreign_export' involving foreign types on the C back-end. It is also needed by the MLDS->C back-end, for cases involving abstract equivalence types which are defined as "float".
• exception_analysis.m annotates each module with information about whether the procedures in the module may throw an exception or not.
• The next pass is termination analysis. The compiler contains two separate termination analysis systems, which are based on different principles.

  The modules involved in the first system are:

  • termination.m is the control module. It sets the argument size and termination properties of builtin and compiler generated procedures, invokes term_pass1.m and term_pass2.m, and writes .trans_opt files and error messages as appropriate.
  • term_pass1.m analyzes the argument size properties of user-defined procedures,
  • term_pass2.m analyzes the termination properties of user-defined procedures.
  • term_traversal.m contains code common to the two passes.
  • term_errors.m defines the various kinds of termination errors and prints the messages appropriate for each.
  • term_util.m defines the main types used in termination analysis and contains utility predicates.
  • post_term_analysis.m contains error checking routines and optimizations that depend upon the information obtained by termination analysis.

  The modules involved in the second system are:

  • term_constr_main.m is the control module; it invokes the others as needed.
  • term_constr_initial.m sets up the initial state of the analysis, based on things such as user-provided annotations.
  • term_constr_build.m builds an abstract representation of the procedures to be analyzed.
  • term_constr_fixpoint.m uses that abstract representation to derive information about the relationships among the sizes of the arguments of each analyzed procedure.
  • term_constr_pass2.m uses this information about argument size relationships to attempt to prove whether the analyzed procedures terminate.
  • term_constr_main_types.m defines the types that represent the result of the analysis.
  • term_constr_data.m defines types needed during the analysis.
  • term_constr_errors.m generates error messages for termination problems.
• trailing_analysis.m pass annotates each module with information about whether the procedures in the module modify the trail or not. This information can be used to avoid redundant trailing operations.
• Minimal model tabling analysis (tabling_analysis.m) annotates each goal in a module with information about whether the goal calls procedures that are evaluated using minimal model tabling. This information can be used to reduce the overhead of minimal model tabling.

The results of these program analyses are written out to both `.opt' and `.trans_opt' files by intermod_analysis.m. `.trans_opt' files consist of nothing but analysis results, while `.opt' files also contain other information written out by intermod.m. This other information falls into two categories.

• the definitions for predicates (exported or local) if these are suitable for other optimizations such as inlining or higher-order specialization, and
• other not-ordinarily-exported parts of the module, such as local type, inst and mode definitions and the declarations of local predicates, if these may be required to make sense of the predicate definitions in the first category.

The code of intermod.m uses intermod_info.m, intermod_decide.m and intermod_status.m as helpers.

The intermod_mark_exported.m module updates the statuses of entities that intermod.m would export. It is used by compiler invocations whose task is something other than the creation of `.opt' files.

Most of the remaining HLDS-to-HLDS transformations are optimizations:

• higher_order.m specializes higher-order and polymorphic predicates in contexts where the value of the relevant higher-order, type_info, and/or typeclass_info arguments are known.
• accumulator.m attempts to introduce accumulators. This optimizes procedures whose task consists of independent associative computations or independent chains of commutative computations by transforming them into a tail recursive form through the introduction of accumulators. If lco (last call optimization, also called last-call-modulo-constructor optimization) is turned on, it can also transform some procedures so that only construction unifications are after the recursive call. This pass must come before lco, unused_args (because eliminating arguments makes it hard to relate the code back to the assertion) and inlining (because it can make the associative call disappear).

  This pass makes use of the goal_store.m module, which is a dictionary-like data structure for storing HLDS goals.

• inlining.m replaces calls with the bodies of the called procedures wherever that transformation looks likely to improve performance.
• loop_inv.m does loop invariant hoisting. This transformation moves computations within loops that are the same on every iteration to the outside of the loop, so that the invariant computations are only computed once. The transformation turns a single looping predicate containing invariant computations into two predicates: one that computes the invariants on the first iteration and then loops by calling the second predicate with extra arguments for the invariant values. This pass should come after inlining, since inlining can expose important opportunities for loop invariant hoisting. Such opportunities might not be visible before inlining because it is possible that only a *part* of the body of a called procedure is loop-invariant.
• The deforestation and partial evaluation pass looks for code that does multiple traversals of data structures within a conjunction, and replaces it with code that avoids creating those intermediate data structures. It also performs loop unrolling where the clause used is known at compile time. Its main module is deforest.m, which makes use of the following submodules. (`pd_' stands for "partial deduction".)
  • constraint.m transforms goals so that goals which can fail are executed earlier.
  • pd_cost.m contains some predicates to estimate the improvement caused by deforest.m.
  • pd_debug.m produces debugging output.
  • pd_info.m contains a state type for deforestation.
  • pd_term.m contains predicates to check that the deforestation algorithm terminates.
  • pd_util.m contains various utility predicates.
• unused_args.m issues warnings about unused arguments from predicates, and create specialized versions without them. Typeinfos are often unused.
• delay_construct.m pushes construction unifications to the right in semidet conjunctions, in an effort to reduce the probability that they will need to be executed.
• unneeded_code.m looks for goals whose results are either not needed at all, or needed in some branches of computation but not others. Provided that the goal in question satisfies some requirements (e.g. it is pure, it cannot fail etc), it either deletes the goal, or moves it to the computation branches where its output is needed.
• lco.m finds predicates whose implementations would benefit from last call optimization modulo constructor application.
• dead_proc_elim.m eliminates dead procedures. Inlining, higher-order specialization and the elimination of unused args can make procedures dead even if the user doesn't, and automatically constructed unification and comparison predicates are often dead as well.
• tupling.m looks for predicates that pass around several arguments, and modifies the code to pass around a single tuple of these arguments instead if this looks like reducing the cost of parameter passing.
• untupling.m does the opposite of tupling.m: it replaces tuple arguments with their components. This can be useful both for finding out how much tupling has already been done manually in the source code, and to break up manual tupling in favor of possibly more profitable automatic tupling.
• dep_par_conj.m transforms parallel conjunctions to add the wait and signal operations required by dependent AND parallelism. To maximize the amount of parallelism available, it tries to push the signals as early as possible in producers and the waits as late as possible in the consumers, creating specialized versions of predicates as needed.
• parallel_to_plain_conj.m transforms parallel conjunctions to plain conjunctions, for use in grades that do not support AND-parallelism.
• granularity.m tries to ensure that programs do not generate too much parallelism. Its goal is to minimize parallelism's overhead while still gaining all the parallelism the machine can actually exploit.
• implicit_parallelism.m is a package whose task is to introduce parallelism into sequential code automatically. Its submodules are
  • introduce_parallelism.m does the main task of the package.
  • push_goals_together.m performs a transformation that allows introduce_parallelism.m to do a better job.
• float_regs.m wraps higher-order terms which use float registers if passed in contexts where regular registers would be expected, and vice versa.
• The rbmm.m subpackage implements region based memory management (usually abbreviated to RBMM), that
  1. analyses programs looking for memory allocations with similar lifetimes, with each such set of memory allocations being grouped into a region,
  2. puts annotates on those allocations that tell the code generator to get the new memory from the allocation's region, and
  3. adds operations to allocate the regions themselves just before their first use, and to deallocate them just after their last use.
  This last step, which deallocates an entire region in a few instructions, is what gives RBMM the potential to be more efficient than traditional garbage collection.

  The RBMM system in the Mercury compiler is experimental, and not ready for general use. The detailed documentation of its component modules awaits the next person to work on it.

The module transform.m contains stuff that is supposed to be useful for high-level optimizations (but which is not yet used).

The last three HLDS-to-HLDS transformations implement term size profiling (size_prof.m and complexity.m) and deep profiling (deep_profiling.m, in the ll_backend.m package). Both passes insert into procedure bodies, among other things, calls to procedures (some of which are impure) that record profiling information.

3-6 The LLDS and MLDS backends

a. LLDS backend

This is the ll_backend.m package.

3a. LLDS-specific HLDS -> HLDS transformations

• saved_vars.m tries to reduce the number of variables that have to be saved across procedure calls. It does this by putting the code that generates the value of a variable just before the use of that variable, duplicating the variable and the code that produces it if necessary, provided the cost of doing so is smaller than the cost of saving and restoring the variable would be.
• stack_opt.m tries to transform procedure definitions to reduce the number of variables that need their own stack slots. The main algorithm in stack_opt.m figures out when variable A can be reached from a cell pointed to by variable B, so that storing variable B on the stack obviates the need to store variable A on the stack as well. This code uses the maximal matching algorithm in matching.m.
• follow_code.m migrates calls following branched structures into each branch, in an effort to improve the results of follow_vars.m (see below).
• After follow_code.m, we invoke simplification again (simplify.m, in the check_hlds.m package). We run this pass a second time in case the intervening transformations have created new opportunities for simplification. It needs to be run immediately before code generation, because it enforces some invariants that the LLDS code generator relies on.
• liveness.m annotates goals with liveness information, which records the birth and death of each variable in the HLDS goal_info.
• stack_alloc.m allocates stack slots to variables with the assistance of the following modules:
  • live_vars.m works out which variables need to be saved on the stack when.
  • graph_colour.m (in the libs.m package) contains the algorithm that stack_alloc.m calls to convert sets of variables that must be saved on the stack at the same time to an assignment of a stack slot to each such variable.
• follow_vars.m traverses backwards over the HLDS, annotating some goals with information about what locations variables will be needed in next. This allows us to generate more efficient code by putting variables in the right spot directly. This module is not called from mercury_compile_llds_back_end.m; it is called from store_alloc.m.
• store_alloc.m annotates each branched goal with a store map, a data structure that maps each variable to the location it should be in when execution reaches the end of that branch. Having all branches putting all variables that are live at the end of the branched control structure into the same location at the end of the branch provides a consistent picture of where things are at the code following the branched control structure.
• goal_path.m (in the check_hlds.m package) optionally annotates every goal with its goal path, which specifies its position in the procedure body. This information is needed by the debugger.

4a. LLDS code generation

• The top level code generation module is proc_gen.m, which looks after the generation of code for procedures (including prologues and epilogues).
• The predicate for generating code for arbitrary goals is in code_gen.m, but that module handles only sequential conjunctions; it calls other modules to handle other kinds of goals.
• ite_gen.m generates code for if-then-elses and negations.
• call_gen.m generates code for both plain calls and generic calls, the latter including higher order calls, class method invocations, calls to events, and casts.
• disj_gen.m generates code for disjunctions.
• par_conj_gen.m generates code for parallel conjunctions.
• unify_gen.m generates code for assign and simple test unifications, but invokes unify_gen_construct.m or unify_gen_deconstruct.m to generate code for construction and deconstruction unifications. All these modules also rely on the utility predicates in unify_gen_test.m and unify_gen_util.m.
• closure_gen.m generates code that creates closures.
• switch_gen.m is the main module for generating code for switches, but for special kinds of switches, it subcontracts the task to the following modules.
  • lookup_switch.m implements switches on integral atomic types (such as integers and enum types) using tables containing the output values of the switch.
  • dense_switch.m implements switches on integral atomic types (such as integers and enum types) using tables containing the address of the code for each switch arm.
  • string_switch.m implements switches on strings using either hash tables or binary search.
  • tag_switch.m implements switches on discriminated union types by switching on first the primary tag and then the secondary tag.
  • switch_case.m contains utility predicates needed to implement all kinds of switches for the LLDS backend.
  • switch_util.m, which is in the backend_libs.m package, also contains utility predicates, but these are needed to implement switches in both the LLDS and MLDS backends.
• commit_gen.m contains code for generating commits.
• pragma_c_gen.m contains code for generating invocations of embedded C code.

The code generator also calls middle_rec.m to do middle recursion optimization, which is implemented during code generation.

The code generation modules above make use of many service modules.

• code_info.m defines the persistent part of the code generator state.
• code_loc_dep.m defines the location-dependent part of the code generator state.
• var_locn.m defines the var_locn type, which is a component of the code_info data structure; it keeps track of the values and locations of variables. It implements eager code generation.
• exprn_aux.m and code_util.m contain various utility predicates.
• lookup_util.m contains some utility predicates that are used in the implementation of both lookup switches and lookup disjunctions.
• continuation_info.m collects information about the live values after calls, for use by the debugger (and in the future, possibly by accurate garbage collection).
• trace_gen.m inserts calls to the runtime debugger.
• trace_params.m (in the libs.m package, since it is considered part of option handling) holds the parameter settings that control the handling of execution tracing.

The purpose of export.m is to allow non-Mercury-generated C code to call Mercury-generated C code. For each `pragma export' declaration, it generates C code fragments which declare and define the C functions which are the interface stubs for procedures exported to C.

The generation of constants for RTTI data structures could also be considered a part of code generation, but for the LLDS back-end this is currently done as part of the output phase (see below).

The result of code generation is the Low Level Data Structure (llds.m), which may also contain some data structures whose types are defined in rtti.m. The code for each procedure is generated as a cord of code fragments which is then flattened.

5a. LLDS transformations

• optimization of jumps to jumps (jumpopt.m)
• elimination of duplicate code sequences within procedures (dupelim.m)
• elimination of duplicate procedure bodies (dupproc.m, invoked directly from mercury_compile_llds_back_end.m)
• optimization of stack frame allocation/deallocation (frameopt.m)
• filling branch delay slots (delay_slot.m)
• dead code and dead label removal (labelopt.m)
• peephole optimization (peephole.m)
• introduction of local C variables (use_local_vars.m)
• removal of redundant assignments, i.e. assignments that assign a value that the target location already holds (reassign.m)

The module opt_debug.m contains utility routines used for debugging these LLDS-to-LLDS optimizations.

Several of these optimizations (frameopt and use_local_vars) also use livemap.m, a module that finds the set of locations live at each label.

Some of the low-level optimization passes use basic_block.m, which defines predicates for converting sequences of instructions to basic block format and back, as well as opt_util.m, which contains miscellaneous predicates for LLDS-to-LLDS optimization.

Use_local_vars numbering also introduces references to temporary variables in extended basic blocks in the LLDS representation of the C code. The transformation to insert the block scopes and declare the temporary variables is performed by wrap_blocks.m.

Depending on which optimization flags are enabled, optimize.m may invoke many of these passes multiple times.

6a. LLDS output

• type_ctor_info.m generates the type_ctor_gen_info structures that list items of information (including unification, index and compare predicates) associated with each declared type constructor that go into the static type_ctor_info data structure. (This module is in the backend_libs.m package, since it is shared with the MLDS back-end.) If the type_ctor_gen_info structure is not eliminated as inaccessible, this module adds the corresponding type_ctor_info structure to the RTTI data structures defined in rtti.m, which are part of the LLDS.
• base_typeclass_info.m (which is also in the backend_libs.m package, since it is also shared with the MLDS back-end) generates the base_typeclass_info structures that list the methods of a class for each instance declaration. These are added to the RTTI data structures, which are part of the LLDS.
• stack_layout.m generates the stack_layout structures that the debugger uses to walk the stack. (In the future, they may also be used for accurate garbage collection.) These structures are created from the data collected in continuation_info.m.

  stack_layout.m uses prog_rep.m to generate bytecode representations of procedure bodies for use by the declarative debugger and the deep profiler, and prog_rep_tables.m to generate the string tables and type tables that these representations use.

• Type_ctor_info structures and stack_layout structures both contain pseudo_type_infos, which are type_infos with holes for type variables. These are generated by pseudo_type_info.m, (which is also in the backend_libs.m package, since it is also shared with the MLDS back-end).
• global_data.m contains a database of the LLDS versions of static terms. If a static term appears several times in the HLDS, it will appear in the LLDS as a single static term that has multiple references to it.
• transform_llds.m is responsible for any source to source transformations on the LLDS which are required to make the C output acceptable to various C compilers. Currently computed gotos can have their maximum size limited to avoid a fixed limit in lcc.
• Final generation of C code is done by the llds_out package. The package subcontracts the output of RTTI structures to rtti_out.m and of other static compiler-generated data structures (such as those used by the debugger, the deep profiler, and in the future by the garbage collector) to layout_out.m.

  The llds_out.m package itself consists of several modules:

  • llds_out_file.m for printing out LLDS modules;
  • llds_out_instr.m for printing out LLDS instructions;
  • llds_out_global.m for printing out C global variables;
  • llds_out_data.m for printing lvals and rvals;
  • llds_out_code_addr.m for printing labels and other code addresses.
  • llds_out_util.m defines some utility types and predicates.

b. MLDS BACK-END

The original LLDS code generator generates very low-level code, since the LLDS was designed to map easily to RISC architectures. The MLDS backend generates much higher-level code, suitable for generating not just C, but also Java and C# (and once upon a time, the intermediate language or IL of the .NET platform). This back-end uses the Medium Level Data Structure (mlds.m) as its intermediate representation. While the LLDS operates on the level of assembly languages (a second generation language, with the first generation being machine code), the MLDS operates on the level of a standard third generation imperative programming language.

3b. MLDS-specific HLDS -> HLDS transformations

• mark_static_terms.m (in the hlds.m package) marks construction unifications which construct static terms. These can be implemented using static constants rather than heap allocation.

When creating the MLDS back-end, we have tried to put into practice the lessons we have learned from writing the LLDS backend. One of these is that we prefer to do things as HLDS to HLDS transformations where possible, since this is much easier to debug initially and to modify later than the alternative approach of doing things inside the HLDS to MLDS code generator. Thus we have these two passes that use HLDS transformations to achieve effects for the MLDS backend that the LLDS backend achieves using code fragments that are scattered all over the LLDS code generator.

• add_trail_ops.m inserts code to manipulate the trail, in particular ensuring that we apply the appropriate trail operations before each choice point, when execution resumes after backtracking, and whenever we do a commit. The trail operations are represented as (and implemented as) calls to impure procedures defined in library/private_builtin.m.
• add_heap_ops.m is very similar to add_trail_ops.m; it inserts code to do heap reclamation on backtracking.

4b. MLDS code generation

• ml_top_gen.m is the top module of the package that converts HLDS code to MLDS. It invokes the other modules of the package as needed.
• ml_proc_gen.m handles the translation of procedures to MLDS.
• ml_code_gen.m handles the tasks common to all kinds of goals, as well as the tasks specific to conjunctions, if-then-elses and negations. For other kinds of goals, ml_code_gen.m invokes other modules.
• ml_unify_gen.m and its subcontractors ml_unify_gen_construct.m and ml_unify_gen_deconstruct.m generate code for unifications with the help of the utility predicates in ml_unify_gen_test.m and ml_unify_gen_util.m.
• ml_closure_gen.m generates code for closures.
• ml_call_gen.m generates code for both plain and generic calls.
• ml_foreign_proc_gen.m generates code that invokes foreign_procs.
• ml_commit_gen.m generates code for commits.
• ml_disj_gen.m generates code for disjunctions.
• ml_switch_gen.m generates code for switches with the help of ml_lookup_switch.m, ml_string_switch.m, ml_tag_switch.m, and ml_simplify_switch.m (which do the same jobs as their LLDS equivalents) and switch_util.m in the backend_libs.m package (since it is also used by LLDS back-end).
• ml_accurate_gc.m handles provisions for accurate garbage collection.

• The main data structure that holds the state of the MLDS code generator is defined in ml_gen_info.m.
• ml_global_data.m looks after the database of global data structures (those created at module scope).
• ml_type_gen.m converts descriptions of HLDS types to MLDS static data.
• type_ctor_info.m and base_typeclass_info.m generate the RTTI data structures defined in rtti.m and pseudo_type_info.m (those four modules are in the backend_libs.m package, since they are shared with the LLDS back-end) and then rtti_to_mlds.m converts these to MLDS.
• The modules ml_args_util.m, ml_code_util.m, ml_target_util.m and ml_util.m provide some general utility predicates.

5b. MLDS transformations

• ml_optimize.m and ml_unused_assigns.m do MLDS to MLDS optimizations.
• ml_elim_nested.m does two MLDS transformations that happen to have a lot in common: (1) eliminating nested functions and (2) adding code to handle accurate garbage collection.
• ml_rename_class.m does what its name suggests: renames classes in the MLDS. It is used by mlds_to_java_file.m to replace long class names with shorter ones.

6b. MLDS output

There are currently three target languages in which we can output MLDS code: C, Java, and C#. (They were added in that order.) Generated C code is intended to be given to a C compiler to turn the .c file into a .o or .pic_o file. Generated Java code is intended to be given to a Java compiler (normally javac) to turn the .java file into a .class file containing Java bytecodes. Generated C# code is intended to be given to a C# compiler to turn the .cs file into a .dll or .exe.

• The various mlds_to_c_*.m modules together write out the MLDS as C code.
• The various mlds_to_java_*.m modules together write out the MLDS as Java code.
• The various mlds_to_cs_*.m modules together write out the MLDS as C# code.
• There is also a module, mlds_dump.m, that dumps out the MLDS in a format designed specifically for debugging.

In more detail:

• The mlds_to_{c,cs,java}_file.m modules write out .c and .h files (for C), .cs files (for C#) and .java files (for Java), calling the other modules below as needed.
• The mlds_to_{c,cs,java}_func.m modules output the declarations and definitions of MLDS functions.
• The mlds_to_{c,cs,java}_stmt.m modules output MLDS statements.
• The mlds_to_{c,cs,java}_data.m modules output MLDS rvals, lvals, and initializers.
• The mlds_to_{c,cs,java}_type.m modules output MLDS types.
• The mlds_to_{c,cs,java}_class.m modules output the definitions of MLDS classes.
• The mlds_to_{c,cs,java}_global.m modules output the declarations and definitions of global variables.
• The mlds_to_{c,cs,java}_export.m modules output code that exports to C, C# or Java (i.e. makes usable from those languages) functions and types defined in Mercury.
• The mlds_to_{c,cs,java}_name.m modules output the names of various MLDS entities.
• The mlds_to_java_wrap.m module creates wrapper classes around methods as way of implementing function pointers.
• The mlds_to_{c,cs,java}_util.m modules contain utility types and predicates used by the other modules above.

The mlds_to_target_util.m module contains types, functions and predicates that are needed by more than one of these MLDS backends.

c. Bytecode backend

This is the bytecode_backend.m package.

The Mercury compiler can translate Mercury programs into bytecode for interpretation by a bytecode interpreter. The intent of this is to achieve faster turn-around time during development. However, the bytecode interpreter has not yet been written.

• bytecode.m defines the internal representation of bytecodes, and contains the predicates to emit them in two forms. The raw bytecode form is emitted into .bytecode files for interpretation, while a human-readable form is emitted into .bytedebug files for visual inspection.
• bytecode_gen.m contains the predicates that translate HLDS into bytecode.
• bytecode_data.m contains the predicates that translate ints, strings and floats into bytecode.

Smart recompilation

This functionality is implemented by the recompilation.m package.

• recompilation.m just includes the recompilation.*.m modules.
• recompilation.item_types.m defines the types that are part of the parse trees of interface files, and the operations we need on them.
• recompilation.record_uses.m defines the data structures we use to record what parts of each imported module the current module uses.
• recompilation.version.m generates version numbers for program items in interface files.
• recompilation.usage.m works out which program items were used during a compilation.
• recompilation.used_file.m writes out that information to a modulename.used file.
• recompilation_check.m is called before recompiling a module. It uses the information written by recompilation.version.m and recompilation.used_file.m to work out whether the recompilation is actually needed.

Miscellaneous

• The modules special_pred.m (in the hlds.m package) and unify_proc.m (in the check_hlds.m package) contain stuff for handling the special compiler-generated predicates which we generate for each type: unify/2, compare/3, and index/1. (Index is used in the implementation of compare/3.)
• dependency_graph.m computes the call graph for a module, and prints it out to a file. (The call graph file is used by the profiler.) The call graph may eventually also be used by det_analysis.m, inlining.m, and other parts of the compiler which could benefit from traversing the predicates in a module in a bottom-up or top-down fashion with respect to the call graph.
• builtin_ops.m defines the types unary_op and binary_op, which are used by all the backends to implement builtin operations.
• c_util.m defines utility routines for generating C code. It is used by both the LLDS and MLDS backends.
• name_mangle.m defines predicates for mangling names to forms acceptable as identifiers in target languages.
• compile_target_code.m invokes compilers and/or linkers for our various target languages to convert the generated code into executables.
• string_encoding.m: defines utility predicates related to string encodings.
• file_util.m contains utility predicates dealing with files, such as searching for a file in a list of directories.
• maybe_util.m defines utility types such as maybe_succeeded (which records the success or failure of an operation) and maybe_changed (which records whether an operation has changed something or not) as well as some operations on those types.
• process_util.m contains predicates that deal with process creation and signal handling. This module is mainly used by make.m and its submodules.
• shell_util.m contains a utility function for quoting arguments to shell commands.
• timestamp.m contains an ADT representing timestamps. It is used by smart recompilation and `mmc --make'.
• graph_colour.m does graph colouring. This is used by the LLDS back-end for register allocation.
• int_emu.m emulates integer operations on values with a given number of bits.
• lp.m implements the linear programming algorithm for optimizing a set of linear constraints on floats with respect to a linear cost function. This is used by the first termination analyser, whose top level is in termination.m.
• lp_rational.m implements the linear programming algorithm for optimizing a set of linear constraints on rational numbers with respect to a linear cost function. This is used by the second, convex-constraint-based termination analyser, whose top level is in term_constr_main.m.
• polyhedron.m implements operations on convex polyhedra. This is used by the second, convex-constraint-based termination analyser, whose top level is in term_constr_main.m.
• rat.m implements rational numbers.
• compiler_util.m contains generic utility predicates, mainly for error handling.
• mmakefiles.m defines a representation for mmakefiles and mmakefile fragments, and predicates for printing them.
• check_libgrades.m checks whether libraries we want to use are installed in the required grade.
• md5.m implements md5 checksums.
• va_map.m implements a map-like interface on top of version_arrays.
• pickle.m provides means to serialise arbitrary data structures into a binary format, from which they can later be restored.
• indent.m provides utilities for dealing with indentation in the text that the compiler writes out to e.g. target language files and HLDS dumps.
• system_cmds.m contains predicates for invoking commands via the shell.
• copy_util.m contains predicates for copying files.

Currently undocumented

• analysis.m
• mmc_analysis.m