Peer to Peer Article
Knowledge Based
SCALA and Metaprogramming Schema
1.1 Metaprogramming by performing some operations at compile time. However, sometimes inlining is not enough and we need more powerful ways to analyze and synthesize programs at compile time. Macros enable us to do exactly this: treat programs as data and manipulate them.
Macros Treat Programs as Values
With a macro, we can treat programs as values, which allows us to analyze and generate them at compile time.
A Scala expression with type T is represented by an instance of the type scala.quoted.Expr[T].
We will dig into the details of the type Expr[T], as well as the different ways of analyzing and constructing instances, when talking about Quoted Code and Reflection. For now, it suffices to know that macros are metaprograms that manipulate expressions of type Expr[T].
The following macro implementation prints the expression of the provided argument at compile-time in the standard output of the compiler process:
import scala.quoted.* // imports Quotes, Expr
def inspectCode(x: Expr[Any])(using Quotes): Expr[Any] =
println(x.show)
x
After printing the argument expression, we return the original argument as a Scala expression of type Expr[Any].
As foreshadowed in the section on Inline, inline methods provide the entry point for macro definitions:
inline def inspect(inline x: Any): Any = ${ inspectCode(‘x) }
All macros are defined with an inline def. The implementation of this entry point always has the same shape:
- they only contain a single splice ${ … }
- the splice contains a single call to the method that implements the macro (for example inspectCode).
- the call to the macro implementation receives the quoted parameters (that is ‘x instead of x) and a contextual Quotes.
We will dig deeper into these concepts later in this and the following sections.
Calling our inspect macro inspect(sys error “abort”) prints a string representation of the argument expression at compile time:
scala.sys.error(“abort”)
Macros and Type Parameters
If the macro has type parameters, the implementation will also need to know about them. Just like scala.quoted.Expr[T] represents a Scala expression of type T, we use scala.quoted.Type[T] to represent the Scala type T.
inline def logged[T](inline x: T): T = ${ loggedCode(‘x) }
def loggedCode[T](x: Expr[T])(using Type[T], Quotes): Expr[T] = …
Both the instance of Type[T] and the contextual Quotes are automatically provided by the splice in the corresponding inline method (that is, logged) and can be used by the macro implementation.
Defining and Using Macros
A key difference between inlining and macros is the way they are evaluated. Inlining works by rewriting code and performing optimisations based on rules the compiler knows. On the other hand, a macro executes user-written code that generates the code that the macro expands to.
Technically, compiling the inlined code ${ inspectCode(‘x) } calls the method inspectCode at compile time (through Java reflection), and the method inspectCode then executes as normal code.
To be able to execute inspectCode, we need to compile its source code first. As a technical consequence, we cannot define and use a macro in the same class/file. However, it is possible to have the macro definition and its call in the same project as long as the implementation of the macro can be compiled first.
1.2 Suspended Files
To allow defining and using macros in the same project, only those calls to macros that have already been compiled are expanded. For all other (unknown) macro calls, the compilation of the file is suspended. Suspended files are only compiled after all non suspended files have been successfully compiled. In some cases, you will have cyclic dependencies that will block the completion of the compilation. To get more information on which files are suspended you can use the -Xprint-suspension compiler flag.
Example: Statically Evaluating power with Macros
Let us recall our definition of power from the section on Inline that specialized the computation of xⁿ for statically known values of n.
inline def power(x: Double, inline n: Int): Double =
inline if n == 0 then 1.0
else inline if n % 2 == 1 then x * power(x, n – 1)
else power(x * x, n / 2)
In the remainder of this section, we will define a macro that computes xⁿ for a statically known values x and n. While this is also possible purely with inline, implementing it with macros will illustrate a few things.
inline def power(inline x: Double, inline n: Int) =
${ powerCode(‘x, ‘n) }
def powerCode(
x: Expr[Double],
n: Expr[Int]
)(using Quotes): Expr[Double] = …
Simple Expressions
We could implement powerCode as follows:
def pow(x: Double, n: Int): Double =
if n == 0 then 1 else x * pow(x, n – 1)
def powerCode(
x: Expr[Double],
n: Expr[Int]
)(using Quotes): Expr[Double] =
val value: Double = pow(x.valueOrAbort, n.valueOrAbort)
Expr(value)
Here, the pow operation is a simple Scala function that computes the value of xⁿ. The interesting part is how we create and look into the Exprs.
Creating Expression From Values
Let’s first look at Expr.apply(value). Given a value of type T, this call will return an expression containing the code representing the given value (that is, of type Expr[T]). The argument value to Expr is computed at compile-time, at runtime we only need to instantiate this value.
Creating expressions from values works for all primitive types, tuples of any arity, Class, Array, Seq, Set, List, Map, Option, Either, BigInt, BigDecimal, StringContext. Other types can also work if a ToExpr is implemented for it, we will see this later.
Extracting Values from Expressions
The second method we use in the implementation of powerCode is Expr[T].valueOrAbort, which has an effect opposite to Expr.apply. It attempts to extract a value of type T from an expression of type Expr[T]. This can only succeed, if the expression directly contains the code of a value, otherwise, it will throw an exception that stops the macro expansion and reports that the expression did not correspond to a value.
Instead of valueOrAbort, we could also use the value operation, which will return an Option. This way we can report the error with a custom error message.
Reporting Custom Error Messages
The contextual Quotes parameter provides a report object that we can use to report a custom error message. Within a macro implementation method, you can access the contextual Quotes parameter with the quotes method (imported with import scala.quoted.*), then import the report object by import quotes.reflect.report.
Providing the Custom Error
We will provide the custom error message by calling errorAndAbort on the report object as follows:
def powerCode(
x: Expr[Double],
n: Expr[Int]
)(using Quotes): Expr[Double] =
import quotes.reflect.report
(x.value, n.value) match
case (Some(base), Some(exponent)) =>
val value: Double = pow(base, exponent)
Expr(value)
case (Some(_), _) =>
report.errorAndAbort(“Expected a known value for the exponent, but was ” + n.show, n)
case _ =>
report.errorAndAbort(“Expected a known value for the base, but was ” + x.show, x)
Alternatively, we can also use the Expr.unapply extractor
…
(x, n) match
case (Expr(base), Expr(exponent)) =>
val value: Double = pow(base, exponent)
Expr(value)
case (Expr(_), _) => …
case _ => …
The operations value, valueOrAbort, and Expr.unapply will work for all primitive types, tuples of any arity, Option, Seq, Set, Map, Either and StringContext. Other types can also work if an FromExpr is implemented for it, we will see this later.
Showing Expressions
In the implementation of inspectCode, we have already seen how to convert expressions to the string representation of their source code using the .show method. This can be useful to perform debugging on macro implementations:
def debugPowerCode(
x: Expr[Double],
n: Expr[Int]
)(using Quotes): Expr[Double] =
println(
s”powerCode \n” +
s” x := ${x.show}\n” +
s” n := ${n.show}”)
val code = powerCode(x, n)
println(s” code := ${code.show}”)
code
Working with Varargs
Varargs in Scala are represented with Seq, hence when we write a macro with a vararg, it will be passed as an Expr[Seq[T]]. It is possible to recover each individual argument (of type Expr[T]) using the scala.quoted.Varargs extractor.
import scala.quoted.* // imports `Varargs`, `Quotes`, etc.
inline def sumNow(inline nums: Int*): Int =
${ sumCode(‘nums) }
def sumCode(nums: Expr[Seq[Int]])(using Quotes): Expr[Int] =
import quotes.reflect.report
nums match
case Varargs(numberExprs) => // numberExprs: Seq[Expr[Int]]
val numbers: Seq[Int] = numberExprs.map(_.valueOrAbort)
Expr(numbers.sum)
case _ => report.errorAndAbort(
“Expected explicit varargs sequence. ” +
“Notation `args*` is not supported.”, nums)
The extractor will match a call to sumNow(1, 2, 3) and extract a Seq[Expr[Int]] containing the code of each parameter. But, if we try to match the argument of the call sumNow(nums*), the extractor will not match.
Varargs can also be used as a constructor. Varargs(Expr(1), Expr(2), Expr(3)) will return an Expr[Seq[Int]]. We will see how this can be useful later.
Complex Expressions
So far, we have only seen how to construct and destruct expressions that correspond to simple values. In order to work with more complex expressions, Scala 3 offers different metaprogramming facilities ranging from
- additional constructors like Expr.apply,
- over quoted pattern matching,
- to a full reflection API;
each increasing in complexity and potentially losing safety guarantees. It is generally recommended to prefer simple APIs over more advanced ones. In the remainder of this section, we introduce some more additional constructors and destructors, while subsequent chapters introduce the more advanced APIs.
Collections
We have seen how to convert a List[Int] into an Expr[List[Int]] using Expr.apply. How about converting a List[Expr[Int]] into an Expr[List[Int]]? We mentioned that Varargs.apply can do this for sequences; likewise, for other collection types, corresponding methods are available:
- Expr.ofList: Transform a List[Expr[T]] into Expr[List[T]]
- Expr.ofSeq: Transform a Seq[Expr[T]] into Expr[Seq[T]] (just like Varargs)
- Expr.ofTupleFromSeq: Transform a Seq[Expr[T]] into Expr[Tuple]
- Expr.ofTuple: Transform a (Expr[T1], …, Expr[Tn]) into Expr[(T1, …, Tn)]
Simple Blocks
The constructor Expr.block provides a simple way to create a block of code { stat1; …; statn; expr }. Its first arguments is a list with all the statements and the second argument is the expression at the end of the block.
inline def test(inline ignore: Boolean, computation: => Unit): Boolean =
${ testCode(‘ignore, ‘computation) }
def testCode(ignore: Expr[Boolean], computation: Expr[Unit])(using Quotes) =
if ignore.valueOrAbort then Expr(false)
else Expr.block(List(computation), Expr(true))
The Expr.block constructor is useful when we want to generate code contanining several side effects. The macro call test(false, EXPRESSION) will generate { EXPRESSION; true}, while the call test(true, EXPRESSION) will result in false.
Simple Matching
The method Expr.matches can be used to check if one expression is equal to another. With this method we could implement an value operation for Expr[Boolean] as follows.
def value(boolExpr: Expr[Boolean]): Option[Boolean] =
if boolExpr.matches(Expr(true)) then Some(true)
else if boolExpr.matches(Expr(false)) then Some(false)
else None
It may also be used to compare two user written expression. Note, that matches only performs a limited amount of normalization and while for instance the Scala expression 2 matches the expression { 2 }, this is not the case for the expression { val x: Int = 2; x }.
Arbitrary Expressions
Last but not least, it is possible to create an Expr[T] from arbitary Scala code by enclosing it in quotes. For example, ‘{ ${expr}; true } will generate an Expr[Boolean] equivalent to Expr.block(List(expr), Expr(true)). The subsequent section on Quoted Code presents quotes in more detail.
1.3 What is a Closure in Scala?
In Scala, a closure refers to a function that captures and retains references to variables from its surrounding lexical scope even after that scope has exited. In other words, it “closes over” or encapsulates the variables, allowing them to persist within the function’s scope. Closures are a fundamental concept in functional programming languages like Scala, and they enable powerful and flexible programming techniques.
Closures have two main characteristics:
- Encapsulation of Variables: Closures can access and manipulate variables that are defined in their containing or enclosing function, even after that function has completed execution. This behavior is possible because the closure retains a reference to those variables.
- Independence: Closures are self-contained units of behavior, meaning they don’t rely on global or external state. Instead, they carry their own encapsulated state, making them predictable and easy to reason about.
Closures are particularly valuable in scenarios where you need to pass behavior as a parameter, such as in functional programming paradigms like map, reduce, or filter operations. They allow you to create functions on-the-fly that can use and modify variables from their enclosing scope, providing a level of flexibility and expressiveness that is essential for functional programming.
In Scala, closures are often used in conjunction with higher-order functions like map, filter, and reduce to process collections of data in a concise and elegant way. These functions take functions (closures) as arguments and apply them to each element of a collection, making it easy to perform operations like transformation, filtering, or aggregation.
A closure in Scala is a function that captures and retains references to variables from its surrounding lexical scope, allowing for powerful and flexible programming techniques. Closures are a fundamental concept in functional programming and are essential for writing expressive and concise code when working with collections and higher-order functions in Scala.
1. 5 Defining a Closure: In Scala, you can define a closure by creating a function that references variables from its outer scope. These variables are captured and retained by the closure. Here’s a simple example:
def outerFunction(x: Int): Int => Int = {
val factor = x
(y: Int) => y * factor // Closure captures ‘factor’
}
In this example, outerFunction takes an x as an argument and returns a function that multiplies its argument by x. The inner function (y: Int) => y * factor is a closure because it captures the factor variable from its surrounding scope. Run the above code in your editor for a better and clear explanation.
2. Using Closures: You can use closures like regular functions. In this case:
val doubler = outerFunction(2)
val result = doubler(5) // This will return 10
Here, doubler is a closure that captures factor from the outer scope, and when invoked with doubler(5), it multiplies 5 by 2 (the captured factor). Run the above code in your editor for a better and clear explanation.
2.5 Closures with Mutable Variables: Closures can also capture mutable variables. However, you need to be cautious with this, as it can lead to unexpected behavior due to shared state:
def counter(): () => Int = {
var count = 0
() => {
count += 1
count
}
}
In this example, the closure captures the mutable count variable. Each time it’s called, it increments count. Run the above code in your editor for a better and clear explanation.
3.0 Closures in Collections: Closures are often used in Scala collections for operations like map, filter, and reduce. For instance:
val numbers = List(1, 2, 3, 4, 5)
val doubled = numbers.map(x => x * 2) // ‘x’ is a closure capturing the current element
Here, the closure (x => x * 2) is passed to map, and it captures each element of the numbers list as it iterates. Run the above code in your editor for a better and clear explanation.
Example
Here’s a simple example of a closure in Scala:
object ClosureExample {
def main(args: Array[String]): Unit = {
val factor = 5 // Variable from the outer scope
// Closure that captures ‘factor’ from the outer scope
val multiplier = (x: Int) => x * factor
// Using the closure
val result1 = multiplier(10) // Multiplies 10 by 5 (factor)
val result2 = multiplier(8) // Multiplies 8 by 5 (factor)
println(s”Result 1: $result1″) // Output: Result 1: 50
println(s”Result 2: $result2″) // Output: Result 2: 40
}
}
Types of Closure Function
Closures in Scala can be categorized into two main types: pure closures and impure closures. These distinctions are fundamental in functional programming, as they describe the behavior of functions concerning side effects and referential transparency.
Pure Closures:
Pure closures, also known as pure functions or referentially transparent functions, adhere to strict functional programming principles. They are characterized by the following key attributes:
- Encapsulation of Behavior and State: A pure closure encapsulates both behavior (in the form of a function) and the variables or data it relies on. This encapsulation ensures that the function’s behavior is context-independent and self-contained. The closure captures its lexical environment, meaning it retains references to variables in its surrounding scope. These captured variables are often referred to as “free variables.”
- Immutable Variables: The variables captured by a pure closure are immutable. Once a variable is assigned a value, it cannot be modified. This immutability guarantees that the closure’s behavior remains consistent throughout its lifetime. Immutability also ensures that there are no unexpected side effects, making the code more predictable and easier to reason about.
- No Side Effects: A pure closure does not produce side effects. Side effects include actions like modifying variables outside the closure’s scope, printing to the console, or performing I/O operations. In pure closures, all effects are contained within the function.
- Referential Transparency: Referential transparency is a crucial concept in functional programming. It means that a function, given the same inputs, will always produce the same output without affecting or being affected by external factors. Pure closures exhibit referential transparency because they rely solely on their captured variables and do not depend on any external state.
- No Dependency on Global State: A pure closure should not rely on global or mutable state. It depends only on the variables captured at the time of its creation. This characteristic ensures that the closure’s behavior remains consistent regardless of changes to global or external state.
- First-Class Citizens: In functional programming languages, functions, including pure closures, are treated as first-class citizens. This means they can be assigned to variables, passed as arguments to other functions, returned as values, and stored in data structures. The ability to treat closures as first-class citizens allows for higher-order functions, which take closures as parameters and enable powerful abstraction and composition.
- Lazy Evaluation: Some functional programming languages implement lazy evaluation, where functions are not evaluated until their results are actually needed. Pure closures can benefit from lazy evaluation, as they don’t introduce side effects when evaluated.
- Deterministic Output: Given the same inputs (captured variables), a pure closure will always produce the same output. This determinism is essential for reasoning about code behavior and for achieving predictability.
- No Dependency on External Context: A pure closure should not rely on external context, such as system time or user input. Its behavior should be solely determined by its inputs (captured variables) and parameters.
- Testability: Pure closures are highly testable because they have no hidden dependencies or side effects. Testing pure closures involves providing specific inputs and asserting that the outputs match the expected results, making it easier to write unit tests.
Implementation
- Deterministic Behavior: The closure (x: Int) => x * x always produces the same output (x * x) for the same input (x). Thus, it has deterministic behavior.
- No Side Effects: The closure does not modify any external variables, perform I/O operations, or have any other side effects. It purely calculates the square of its input.
- Immutable Data: It operates on immutable data (x is an immutable integer), and it does not modify the input.
- Referential Transparency: The closure is referentially transparent because you can replace a call to square(x) with its result (x * x) without changing the program’s behavior.
Example:
Here’s an example of a pure closure in Scala:
def add(a: Int, b: Int): Int = a + b
The add function is pure because it takes two integers as inputs and returns their sum, without any side effects or reliance on external state. You can safely replace add(2, 3) with 5 anywhere in your code without changing the program’s behavior. Run the above code in your editor for a better and clear explanation.
Impure Closures:
Impure closures, in contrast, do not strictly adhere to the principles of functional programming. They exhibit the following characteristics:
- Access to External State: Impure closures can access variables or data from their enclosing scope, but they often depend on variables outside of their lexical environment. This means they can introduce unexpected behavior if the external state changes.
- Modifying External State: Unlike pure closures, which are typically side-effect-free, impure closures may modify external state. They can change the values of variables or data outside of their own scope, leading to unintended consequences in a program.
- Non-Deterministic Behavior: Impure closures can exhibit non-deterministic behavior because their results depend on external factors. The same impure closure may produce different outcomes when executed at different points in time due to changes in external state.
- Limited Reusability: Impure closures are often less reusable than pure closures. They may have hidden dependencies on external state that make them specific to certain contexts, making it challenging to use them in different scenarios.
- Debugging Complexity: Debugging impure closures can be more complex. When an impure closure exhibits unexpected behavior, identifying the cause may involve tracing changes in external state, which can be challenging in large codebases.
- Concurrency Challenges: Impure closures can lead to race conditions and other concurrency-related issues when multiple threads or processes access and modify the same external state concurrently. Managing shared state in concurrent environments can be error-prone.
- Difficulty in Testing: Testing impure closures can be more challenging. To ensure their behavior is correct, you need to set up and manage the external state accurately, which can be complex and error-prone in itself.
- Reduced Predictability: The use of impure closures can reduce the predictability and understandability of code. It becomes more challenging to determine how changes in one part of the program affect the behavior of impure closures elsewhere.
Implementation of impure closure function in scala
- Declare Mutable State: Start by declaring a mutable variable or state that the impure closure will modify. This mutable state will be accessible from within the closure.
- Create the Impure Closure: Define a function or closure that exhibits impure characteristics. This function should perform some side effects or rely on the mutable state you declared earlier.
- Usage of the Impure Closure: Utilize the impure closure in your code as needed. Keep in mind that invoking the closure will result in side effects and potentially modify the external mutable state.
- Exercise Caution and Documentation: When working with impure closures, exercise caution and be aware of their side effects and mutable state changes. It’s essential to document the behavior of impure closures to ensure that their usage is well-understood and controlled.
Impure closures are useful in specific scenarios where side effects or mutable state management is required, such as I/O operations or performance optimizations.
Example:
Here’s an example of implementing an impure closure function:
object ImpureClosureExample {
var total = 0 // Mutable state
// Impure closure function that adds a value to ‘total’ and returns it
val addToTotal: Int => Int = (x: Int) => {
total += x // Modifies external mutable ‘total’ state
total // Returns the modified ‘total’
}
def main(args: Array[String]): Unit = {
// Using the impure closure function
val result1 = addToTotal(5) // Result is 5
val result2 = addToTotal(3) // Result is 8
println(s”Result 1: $result1″) // Output: Result 1: 5
println(s”Result 2: $result2″) // Output: Result 2: 8
}
}
In this example, we have created an impure closure function named addToTotal. Run the above code in your editor for a better and clear explanation. Here’s how it exhibits impure characteristics:
- Side Effects: The addToTotal closure has side effects because it modifies the external mutable variable total as a side effect. When you call addToTotal(5), it adds 5 to the total, and when you call addToTotal(3), it further modifies the total to 8.
- Mutable State: The variable total is mutable, and the closure modifies it, changing its value over time. This mutable state introduces non-deterministic behavior because the result of calling addToTotal depends on the current state of total.
- Lack of Referential Transparency: This impure closure lacks referential transparency. You cannot safely replace a call to addToTotal(5) with its result because doing so would skip the side effect of updating total.
Conclusion
- Pure and Impure: Closures can be categorized as pure or impure. Pure closures adhere to functional principles, ensuring deterministic and side-effect-free behavior, while impure closures introduce side effects and may rely on mutable state.
- Side Effects and Mutability: Impure closures are essential for tasks involving side effects, I/O operations, or mutable state management. However, they come with increased complexity and require careful handling.
- Optimizations: Pure closures facilitate code optimization and parallelization, as their behavior is solely determined by their inputs, allowing for more efficient execution.
- Modularity and Reusability: Closures enhance code modularity and promote the creation of reusable components by encapsulating behavior and data.
- Documentation and Care: Regardless of closure type, it’s crucial to document their behavior, especially when working with impure closures. Careful consideration of when and where to use closures ensures code clarity and mainta