prestoadmin package¶

Subpackages¶

Submodules¶

prestoadmin.catalog module¶

Module for presto catalog configurations

prestoadmin.catalog.add[source]¶

Deploy configuration for a catalog onto a cluster.

E.g.: ‘presto-admin catalog add tpch’ deploys a configuration file for the tpch connector. The configuration is defined by tpch.properties in the local catalog directory, which defaults to ~/.prestoadmin/catalog.

If no catalog name is specified, then configurations for all catalogs in the catalog directory will be deployed

Parameters:	- Name of the catalog to be added (name) –

prestoadmin.catalog.remove[source]¶

Remove a catalog from the cluster.

Parameters:	- Name of the catalog to be removed (name) –

prestoadmin.collect module¶

Module for gathering various debug information for incident reporting using presto-admin

prestoadmin.collect.logs[source]¶: Gather all the server logs and presto-admin log and create a tar file.

prestoadmin.collect.query_info[source]¶

Gather information about the query identified by the given query_id and store that in a JSON file.

Parameters:	- id of the query for which info has to be gathered (query_id) –

prestoadmin.collect.system_info[source]¶: Gather system information like nodes in the system, presto version, presto-admin version, os version etc.

prestoadmin.config module¶

Module for reading, writing, and processing configuration files

prestoadmin.config.fill_defaults(conf, defaults)[source]¶

prestoadmin.config.get_conf_from_config_file(path)[source]¶

prestoadmin.config.get_conf_from_json_file(path)[source]¶

prestoadmin.config.get_conf_from_properties_data(data)[source]¶

prestoadmin.config.get_conf_from_properties_file(path)[source]¶

prestoadmin.config.json_to_string(conf)[source]¶

prestoadmin.config.split_to_pair(line)[source]¶

prestoadmin.config.write(output, path)[source]¶

prestoadmin.config.write_conf_to_file(conf, path)[source]¶

prestoadmin.config.write_config_file(conf, path)[source]¶

prestoadmin.config.write_properties_file(conf, path)[source]¶

prestoadmin.configure_cmds module¶

Module for various configuration management tasks using presto-admin

prestoadmin.configure_cmds.deploy[source]¶

Deploy configuration on the remote hosts.

Possible arguments are -: coordinator - Deploy the coordinator configuration to the coordinator node workers - Deploy workers configuration to the worker nodes. This will not deploy configuration for a coordinator that is also a worker

If no rolename is specified, then configuration for all roles will be deployed. If there is no presto configuration file found in the configuration directory, default files will be deployed

Parameters:	- [coordinator\|workers] (rolename) –

prestoadmin.configure_cmds.show[source]¶

Print to the user the contents of the configuration files deployed

If no config_type is specified, then all four configurations will be printed. No warning will be printed for a missing log.properties since it is not a required configuration file.

Parameters:	config_type – [node\|jvm\|config\|log]

prestoadmin.coordinator module¶

Module for the presto coordinator’s configuration. Loads and validates the coordinator.json file and creates the files needed to deploy on the presto cluster

class prestoadmin.coordinator.Coordinator[source]¶

Bases: prestoadmin.node.Node

DEFAULT_PROPERTIES = {'jvm.config': ['-server', '-Xmx16G', '-XX:-UseBiasedLocking', '-XX:+UseG1GC', '-XX:G1HeapRegionSize=32M', '-XX:+ExplicitGCInvokesConcurrent', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:+UseGCOverheadLimit', '-XX:+ExitOnOutOfMemoryError', '-XX:ReservedCodeCacheSize=512M', '-DHADOOP_USER_NAME=hive'], 'config.properties': {'query.max-memory': '50GB', 'coordinator': 'true', 'discovery-server.enabled': 'true', 'http-server.http.port': '8080', 'node-scheduler.include-coordinator': 'false', 'query.max-memory-per-node': '8GB'}, 'node.properties': {'node.launcher-log-file': '/var/log/presto/launcher.log', 'node.server-log-file': '/var/log/presto/server.log', 'catalog.config-dir': '/etc/presto/catalog', 'node.data-dir': '/var/lib/presto/data', 'node.environment': 'presto', 'plugin.dir': '/usr/lib/presto/lib/plugin'}}¶

default_config(filename)[source]¶

static validate(conf)[source]¶

prestoadmin.deploy module¶

Common module for deploying the presto configuration

prestoadmin.deploy.configure_presto(conf, remote_dir)[source]¶

prestoadmin.deploy.coordinator()[source]¶: Deploy the coordinator configuration to the coordinator node

prestoadmin.deploy.deploy(confs, remote_dir)[source]¶

prestoadmin.deploy.deploy_node_properties(content, remote_dir)[source]¶

prestoadmin.deploy.dict_to_equal_format(conf)[source]¶

prestoadmin.deploy.escape_single_quotes(text)[source]¶

prestoadmin.deploy.key_val_to_equal(items)[source]¶

prestoadmin.deploy.list_to_line_separated(conf)[source]¶

prestoadmin.deploy.output_format(conf)[source]¶

prestoadmin.deploy.secure_create_directory(filepath, user_group, mode=755)[source]¶

prestoadmin.deploy.secure_create_file(filepath, user_group, mode=600)[source]¶

prestoadmin.deploy.workers()[source]¶: Deploy workers configuration to the worker nodes. This will not deploy configuration for a coordinator that is also a worker

prestoadmin.deploy.write_to_remote_file(text, filepath, owner, mode=600)[source]¶

prestoadmin.fabric_patches module¶

Monkey patches needed to change logging and error handling in Fabric

prestoadmin.fabric_patches.abort(msg)[source]¶

prestoadmin.fabric_patches.execute(task, *args, **kwargs)[source]¶: Patched version of fabric’s execute task with alternative error handling

prestoadmin.fabric_patches.log_output(out)[source]¶

prestoadmin.fabric_patches.run(*args, **kwargs)[source]¶

prestoadmin.fabric_patches.sudo(*args, **kwargs)[source]¶

prestoadmin.fabric_patches.warn(msg)[source]¶

prestoadmin.file module¶

Commands for running scripts on a cluster

prestoadmin.file.run[source]¶

Run an arbitrary script on all nodes in the cluster.

Parameters:	- The path to the script (script) – - Where to put the script on the cluster. Default is /tmp. (remote_dir) –

prestoadmin.file.copy[source]¶

Copy a file to all nodes in the cluster.

Parameters:	- The path to the file (local_file) – - Where to put the file on the cluster. Default is /tmp. (remote_dir) –

prestoadmin.main module¶

This module contains Fab’s main method plus related subroutines.

main is executed as the command line fab program and takes care of parsing options and commands, loading the user settings file, loading a fabfile, and executing the commands given.

The other callables defined in this module are internal only. Anything useful to individuals leveraging Fabric as a library, should be kept elsewhere.

prestoadmin.main.display_command(name, code=0)[source]¶: Print command function’s docstring, then exit. Invoked with -d/–display.

prestoadmin.main.extract_tasks(imported_vars)[source]¶: Handle extracting tasks from a given list of variables

prestoadmin.main.get_default_options(options, non_default_options)[source]¶

Given a dictionary of options containing the defaults optparse has filled in, and a dictionary of options containing only options parsed from the command line, returns a dictionary containing the default options that remain after removing the default options that were overridden by the options passed on the command line.

Mathematically, this returns a dictionary with default_options.keys = options.keys() non_default_options.keys() where is the set difference operator. The value of a key present in default_options is the value of the same key in options.

prestoadmin.main.get_task_docstring(task)[source]¶

prestoadmin.main.invalid_command_error(arguments)[source]¶

prestoadmin.main.is_classic_task(tup)[source]¶: Takes (name, object) tuple, returns True if it’s a non-Fab public callable.

prestoadmin.main.is_task_module(a)[source]¶: Determine if the provided value is a task module

prestoadmin.main.is_task_object(a)[source]¶

Determine if the provided value is a Task object.

This returning True signals that all tasks within the fabfile module must be Task objects.

prestoadmin.main.list_commands(docstring, format_)[source]¶

Print all found commands/tasks, then exit. Invoked with -l/--list.

If docstring is non-empty, it will be printed before the task list.

format_ should conform to the options specified in LIST_FORMAT_OPTIONS, e.g. "short", "normal".

prestoadmin.main.load_config(load_config_callback)[source]¶: This provides a patch point for the unit tests so that individual test cases don’t need to know the internal details of what happens in _load_topology. See test_main.py for examples.

prestoadmin.main.load_fabfile(path, importer=None)[source]¶

Import given fabfile path and return (docstring, callables).

Specifically, the fabfile’s __doc__ attribute (a string) and a dictionary of {'name': callable} containing all callables which pass the “is a Fabric task” test.

prestoadmin.main.load_tasks_from_module(imported)[source]¶: Handles loading all of the tasks for a given imported module

prestoadmin.main.main(*args, **kwargs)[source]¶: Main command-line execution loop.

prestoadmin.main.parse_and_validate_commands(args=['-b', 'html', '-W', '-d', '_build/doctrees', '.', '_build/html'])[source]¶

prestoadmin.main.parse_arguments(arguments, commands)[source]¶

Parse string list into list of tuples: command, args.

commands is formatted like {‘install’ : {‘server’ : WrappedCallable, ‘cli’ : WrappedCallable}, ‘topology’: {‘show’ : WrappedCallable}}

Thus, since our arguments are separated by spaces, and is of the form [‘install’, ‘server’], we iterate through the commands, progressively going deeper into the dict. If we run out of elements in the dict, the rest of the tokens are arguments to the function. If we don’t get down to the bottom-most level, the command is not valid. If at any point the next token is not in the possible_cmd map, the command is invalid.

prestoadmin.main.parser_for_options()[source]¶

Handle command-line options with LoggingOptionParser.

Return parser, largely for use in parse_arguments.

On this parser, you must call parser.parse_args()

prestoadmin.main.run_tasks(task_list)[source]¶

prestoadmin.main.show_commands(docstring, format, code=0)[source]¶

prestoadmin.main.update_output_levels(show, hide)[source]¶

Update state.output values as per given comma-separated list of key names.

For example, update_output_levels(show='debug,warnings') is functionally equivalent to state.output['debug'] = True ; state.output['warnings'] = True. Conversely, anything given to hide sets the values to False.

prestoadmin.main.validate_hosts(cli_hosts, config_path)[source]¶

prestoadmin.mode module¶

Module for handling presto-admin mode-related functionality.

prestoadmin.mode.for_mode(mode, mode_map)[source]¶

prestoadmin.mode.get[source]¶: Display the current mode.

prestoadmin.mode.get_mode(validate=True)[source]¶

prestoadmin.mode.list[source]¶: List the supported modes.

prestoadmin.mode.select[source]¶: Change the mode.

prestoadmin.mode.validate_mode(mode)[source]¶

prestoadmin.node module¶

Module for the presto coordinator’s configuration. Loads and validates the coordinator.json file and creates the files needed to deploy on the presto cluster

class prestoadmin.node.Node[source]¶

Bases: object

build_all_defaults()[source]¶

default_config(filename)[source]¶

get_conf()[source]¶

static validate(conf)[source]¶

prestoadmin.package module¶

Module for rpm package deploy and install using presto-admin

prestoadmin.package.install[source]¶

Install the rpm package on the cluster

Parameters:	local_path – Absolute path to the rpm to be installed --nodeps (optional) – Flag to indicate if rpm install should ignore checking package dependencies. Equivalent to adding –nodeps flag to rpm -i.

prestoadmin.package.uninstall[source]¶

Uninstall the rpm package from the cluster

Parameters:	rpm_name – Name of the rpm to be uninstalled --nodeps (optional) – Flag to indicate if rpm uninstall) should ignore checking package dependencies. Equivalent to adding –nodeps flag to rpm -e. --force (optional) – Flag to indicate that rpm uninstall should not fail if package is not installed.

prestoadmin.plugin module¶

module for tasks relating to presto plugins

prestoadmin.plugin.add_jar[source]¶

Deploy jar for the specified plugin to the plugin directory.

Parameters:	- Local path to the jar to be deployed (local_path) – - Name of the plugin subdirectory to deploy jars to (plugin_name) – - (Optional) The plugin directory. If no directory is (plugin_dir) – given, ‘/usr/lib/presto/lib/plugin’ is used by default.

prestoadmin.presto_conf module¶

Module for processing presto configuration files

prestoadmin.presto_conf.get_presto_conf(conf_dir)[source]¶

prestoadmin.presto_conf.validate_presto_conf(conf)[source]¶

prestoadmin.prestoclient module¶

Simple client to communicate with a Presto server.

class prestoadmin.prestoclient.PrestoClient(server, user, coordinator_config=None)[source]¶

close()[source]¶

run_sql(sql, schema='default', catalog='hive')[source]¶

Execute a query connecting to Presto server using passed parameters.

Parameters:	sql – SQL query to be executed schema – Presto schema to be used while executing query (default=default) catalog – Catalog to be used by the server
Returns:	list of rows or None if client was unable to connect to Presto

prestoadmin.server module¶

Module for installing, monitoring, and controlling presto server using presto-admin

prestoadmin.server.install[source]¶

Copy and install the presto-server rpm to all the nodes in the cluster and configure the nodes.

The topology information will be read from the config.json file. If this file is missing, then the coordinator and workers will be obtained interactively. Install will fail for invalid json configuration.

The catalog configurations will be read from the local catalog directory which defaults to ~/.prestoadmin/catalog. If this directory is missing or empty then no catalog configuration is deployed.

Install will fail for incorrectly formatted configuration files. Expected format is key=value for .properties files and one option per line for jvm.config

Parameters:

- String specifying location of presto rpm to copy and install (rpm_specifier) –
to nodes in the cluster. The string can specify a presto rpm in the following ways:
1. ‘latest’ to download the latest release
2. Url to download
3. Version number to download
4. Path to a local copy
If rpm_specifier matches multiple forms, it is interpreted as the form with highest precedence. The forms are listed from highest to lowest precedence (going top to bottom) For example, if the rpm_specifier matches the criteria to be a url to download, it will be interpreted as such and will never be interpreted as a version number or a local path.

Before downloading an rpm, install will attempt to find a local copy with a matching version number to the requested rpm. If such a match is found, it will use the local copy instead of downloading the rpm again.
- (optional) Flag to indicate if server install (–nodeps) – should ignore checking Presto rpm package dependencies. Equivalent to adding –nodeps flag to rpm -i.

prestoadmin.server.uninstall[source]¶

Uninstall Presto after stopping the services on all nodes

Parameters:	- (optional) Flag to indicate if server uninstall (–nodeps) – should ignore checking Presto rpm package dependencies. Equivalent to adding –nodeps flag to rpm -e.

prestoadmin.server.upgrade[source]¶

Copy and upgrade a new presto-server rpm to all of the nodes in the cluster. Retains existing node configuration.

The existing topology information is read from the config.json file. Unlike install, there is no provision to supply topology information interactively.

The existing cluster configuration is collected from the nodes on the cluster and stored on the host running presto-admin. After the presto-server packages have been upgraded, presto-admin pushes the collected configuration back out to the hosts on the cluster.

Note that the configuration files in the presto-admin configuration directory are not updated during upgrade.

:param new_rpm_path - The path to the new Presto RPM to: install
:param local_config_dir - (optional) Directory to store the cluster: configuration in. If not specified, a temp directory is used.
:param overwrite - (optional) if set to True then existing: configuration will be orerwriten.
:param –nodeps - (optional) Flag to indicate if server upgrade: should ignore checking Presto rpm package dependencies. Equivalent to adding –nodeps flag to rpm -U.

prestoadmin.server.start[source]¶

Start the Presto server on all nodes

A status check is performed on the entire cluster and a list of servers that did not start, if any, are reported at the end.

prestoadmin.server.stop[source]¶: Stop the Presto server on all nodes

prestoadmin.server.restart[source]¶

Restart the Presto server on all nodes.

A status check is performed on the entire cluster and a list of servers that did not start, if any, are reported at the end.

prestoadmin.server.status[source]¶: Print the status of presto in the cluster

prestoadmin.topology module¶

Module for setting and validating the presto-admin config

prestoadmin.topology.get_conf_from_fabric()[source]¶

prestoadmin.topology.show[source]¶: Shows the current topology configuration for the cluster (including the coordinators, workers, SSH port, and SSH username)

prestoadmin.workers module¶

Module for the presto worker`’s configuration. Loads and validates the workers.json file and creates the files needed to deploy on the presto cluster

class prestoadmin.workers.Worker[source]¶