Implement Drill Trigonometric functions – Using Freemarker code generation
This post is a followup to this last post Contributing to Apache Drill – Math Functions. Lot of Drill function implementations are moving towards Freemarker code gen implementation since there is a need to generate lot of function definitions with different argument datatypes etc.
Freemarker allows us to define templates which would be used to bake java code ready to be used in Apache Drill.
So here are the steps to start contributing to Drill:
Freemarker Textual data description (Tdd):
Some data is prepared for the trigo functions which can be used in the freemarker templates to generate java code. Here we provide all the different implementations of the functions we need freemarker to generate . We have added three definitions for Sin/Cos/Tan functions and have also mentioned all the parameters the function can accept.
The data description for trigo functions is written in : exec/java-exec/src/main/codegen/data/MathFunc.tdd
trigoMathFunctions : [
{className: "Sin", funcName: "sin", javaFunc : "java.lang.Math.sin", outputType: "Float8", types: [
{input: "Int"},
{input: "BigInt"},
{input: "Float4"},
{input: "Float8"},
{input: "SmallInt"},
{input: "TinyInt"},
{input: "UInt1"},
{input: "UInt2"},
{input: "UInt4"},
{input: "UInt8"}
]
},
{className: "Cos", funcName: "cos", javaFunc : "java.lang.Math.cos", outputType: "Float8", types: [
{input: "Int"},
{input: "BigInt"},
{input: "Float4"},
{input: "Float8"},
{input: "SmallInt"},
{input: "TinyInt"},
{input: "UInt1"},
{input: "UInt2"},
{input: "UInt4"},
{input: "UInt8"}
]
},
{className: "Tan", funcName: "tan", javaFunc : "java.lang.Math.tan", outputType: "Float8", types: [
{input: "Int"},
{input: "BigInt"},
{input: "Float4"},
{input: "Float8"},
{input: "SmallInt"},
{input: "TinyInt"},
{input: "UInt1"},
{input: "UInt2"},
{input: "UInt4"},
{input: "UInt8"}
]
}
}
Template for Math Function Generation:
Once the tdd is ready we can now write the Freemarker template to generate our java code. The template would use the data we created in the tdd file. The template for the trigo function is present in: exec/java-exec/src/main/codegen/templates/MathFunctions.java
Refer: Freemarker Manual for help regarding syntax.
<@pp.changeOutputFile name="/org/apache/drill/exec/expr/fn/impl/TrigoMathFunctions.java" />
<#include "/@includes/license.ftl" />
//////////////////////////////////////////////////////////////////////////////////////////////////
//Functions for Trigo Math Functions
//////////////////////////////////////////////////////////////////////////////////////////////////
package org.apache.drill.exec.expr.fn.impl;
import org.apache.drill.common.types.TypeProtos;
import org.apache.drill.common.types.Types;
import org.apache.drill.common.types.TypeProtos.MinorType;
import org.apache.drill.exec.expr.DrillSimpleFunc;
import org.apache.drill.exec.expr.annotations.FunctionTemplate;
import org.apache.drill.exec.expr.annotations.FunctionTemplate.FunctionScope;
import org.apache.drill.exec.expr.annotations.FunctionTemplate.FunctionScope;
import org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
import org.apache.drill.exec.expr.annotations.Output;
import org.apache.drill.exec.expr.annotations.Param;
import org.apache.drill.exec.expr.annotations.Workspace;
import org.apache.drill.exec.expr.fn.impl.StringFunctions;
import org.apache.drill.exec.expr.holders.*;
import org.apache.drill.exec.record.RecordBatch;
/*
* This class is automatically generated from MathFunc.tdd using FreeMarker.
*/
@SuppressWarnings("unused")
public class TrigoMathFunctions{
static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(TrigoMathFunctions.class);
private TrigoMathFunctions(){}
<#list mathFunc.trigoMathFunctions as func>
<#list func.types as type>
@FunctionTemplate(name = "${func.funcName}", scope = FunctionScope.SIMPLE, nulls = NullHandling.NULL_IF_NULL)
public static class ${func.className}${type.input} implements DrillSimpleFunc {
@Param ${type.input}Holder in;
@Output ${func.outputType}Holder out;
public void setup(RecordBatch b) {
}
public void eval() {
out.value = ${func.javaFunc}(in.value);
}
}
</#list>
</#list>
}
Build Drill
$ mvn clean install -DskipTests
Thats All. Its this simple.
We just need to build Drill again and we can check the new java classes generated in the project. The new generated code can be viewed after clean build at : exec/expr/fn/impl/TrigoMathFunctions.java
Output
Time to hit the query on Sqlline and check out output
0: jdbc:drill:zk=local> SELECT employee_id, first_name, . . . . . . . . . . . > sin(employee_id) as SIN_VAL, . . . . . . . . . . . > cos(employee_id) as COS_VAL, . . . . . . . . . . . > tan(employee_id) as TAN_VAL . . . . . . . . . . . > FROM cp.`employee.json` limit 20; +-------------+------------+--------------------+--------------------+--------------------+ | employee_id | first_name | SIN_VAL | COS_VAL | TAN_VAL | +-------------+------------+--------------------+--------------------+--------------------+ | 1 | Sheri | 0.8414709848078965 | 0.5403023058681398 | 1.5574077246549023 | | 2 | Derrick | 0.9092974268256817 | -0.4161468365471424 | -2.185039863261519 | | 4 | Michael | -0.7568024953079282 | -0.6536436208636119 | 1.1578212823495777 | | 5 | Maya | -0.9589242746631385 | 0.28366218546322625 | -3.380515006246586 | | 6 | Roberta | -0.27941549819892586 | 0.9601702866503661 | -0.29100619138474915 | | 7 | Rebecca | 0.6569865987187891 | 0.7539022543433046 | 0.8714479827243187 | | 8 | Kim | 0.9893582466233818 | -0.14550003380861354 | -6.799711455220379 | | 9 | Brenda | 0.4121184852417566 | -0.9111302618846769 | -0.45231565944180985 | | 10 | Darren | -0.5440211108893698 | -0.8390715290764524 | 0.6483608274590866 | | 11 | Jonathan | -0.9999902065507035 | 0.004425697988050785 | -225.95084645419513 | | 12 | Jewel | -0.5365729180004349 | 0.8438539587324921 | -0.6358599286615808 | | 13 | Peggy | 0.4201670368266409 | 0.9074467814501962 | 0.4630211329364896 | | 14 | Bryan | 0.9906073556948704 | 0.1367372182078336 | 7.2446066160948055 | | 15 | Walter | 0.6502878401571168 | -0.7596879128588213 | -0.8559934009085188 | | 16 | Peggy | -0.2879033166650653 | -0.9576594803233847 | 0.3006322420239034 | | 17 | Brenda | -0.9613974918795568 | -0.27516333805159693 | 3.49391564547484 | | 18 | Daniel | -0.7509872467716762 | 0.6603167082440802 | -1.1373137123376869 | | 19 | Dianne | 0.14987720966295234 | 0.9887046181866692 | 0.15158947061240008 | | 20 | Beverly | 0.9129452507276277 | 0.40808206181339196 | 2.237160944224742 | | 21 | Pedro | 0.8366556385360561 | -0.5477292602242684 | -1.5274985276366035 | +-------------+------------+--------------------+--------------------+--------------------+ 20 rows selected (1.971 seconds)
Code
Playtime.
The entire code for the codegen implementation of TrigoMathFunctions can be picked from: https://reviews.apache.org/r/20486/
Feel free to reach out in case of any issues/doubts.
Good Luck Drilling!! Cheers!!
Nice post! Look forward for more posts and contribution 🙂
Thanks Tim.