Spark and system configurations

The configuration of the Java properties and the Spark environment can be done by getting the singleton instance of the configuration class as follows:

conf = gl.get_configuration()

Follows the description of this object:

class Configuration

Class containing all the information regarding the system environment and the Spark environment

set_app_name(name)

Sets the name of the application in spark, By default it is called “gmql_api”

Parameters:name – string Returns:None

set_master(master)

Set the master of the spark cluster By default it is “local[*]”

Parameters:master – string Returns:None

set_spark_conf(key=None, value=None, d=None)

Sets a spark property as a (‘key’, ‘value’) pair of using a dictionary {‘key’: ‘value’, …}

Parameters:

• key – string
• value – string
• d – dictionary

Returns:

None

set_system_conf(key=None, value=None, d=None)

Sets a java system property as a (‘key’, ‘value’) pair of using a dictionary {‘key’: ‘value’, …}

Parameters:

• key – string
• value – string
• d – dictionary

Returns:

None