Debugging and Logging for a YARN-Based ClusterΒΆ
Once the YARN application is launched, you can monitor the status at YARN ResourceManager WebUI.
A successfully launched application will be in RUNNING
state. The
YARN ApplicationMaster UI (eg:
http://master:8088/cluster/app/application_<id>
) will show
slider-appmaster, COORDINATOR and WORKER components and the
associated containers running based on your configuration. You can
also use Slider cli script to Check the Status.
Slider retries to launch Presto in case of failure in the same YARN application.
The YARN application will be still in RUNNING
state during this retry phase.
It ultimately kills the job after 5 unsuccessful retrials.
If you have used Using YARN label your COORDINATOR and WORKER components will be running on nodes which were ‘labeled’.
If you have not used labels, then you can check the status either at
the YARN ResourceManager (eg:
http://master:8088/cluster/app/application_<id>
) or you can use
Check the Status to get the “live” containers, and thus get the
node hosting the Presto components.
If Presto is up and running, then a pgrep
of PrestoServer on your
NodeManager nodes will give you the process details. This should also
give the directory Presto is installed and the configuration files
used by Presto.
If the YARN application has failed to launch Presto, then you may want to
take a look at the slider logs created under YARN log directory for the
corresponding application. It is recommended that log aggregation of YARN application log
files be enabled in YARN, using
yarn.log-aggregation-enable property
in your yarn-site.xml
.
Then slider logs created during the launch of Presto-YARN will be
available locally on your nodemanager nodes (where slider-appmaster
and Presto components-COORDINATOR/WORKER are deployed) under
contanier logs directory eg:
/var/log/hadoop-yarn/application_<id>/container_<id>/
. For any
retries attempted by Slider to launch Presto a new container will be
launched and hence you will find a new container_<id>
directory.
You can look for any errors under errors_*.txt
there, and also
there is a slider-agent.log
file which will give you Slider
application lifetime details.
Subsequently every Slider application owner has the flexibility to
set the include and exclude patterns of file names that they intend
to aggregate, by adding the following properties in their
resources.json
. For example, using
"global": {
"yarn.log.include.patterns": "*",
"yarn.log.exclude.patterns": "*.*out"
}
See http://slider.incubator.apache.org/docs/configuration/resources.html#logagg for details.
If there are no errors in slider.log
then you may want to look at Presto
logs for any errors. Presto logs will be available under the standard Presto data
directory location. By default it is /var/lib/presto/data/var/log
directory where /var/lib/presto/data
is the default data
directory site.global.data_dir
configured in Slider appConfig.json
.
You can find both server.log
and http-request.log
files here. Please note that
log rotation of these Presto log files will have to be manually
enabled (for eg: using
http://linuxcommand.org/man_pages/logrotate8.html)
Presto configuration files will be at /var/lib/presto/etc
directory if you are using the default appConfig.json
property
site.global.config_dir
. The configuration files here will be
generated by Slider and overwritten for every application restart.
These files should NOT be modified manually.