Setting up the dashboard may require some configuration depending on your use model and cluster environment. Integrations with Prometheus and Grafana are optional for extending visualization capabilities.
Port forwarding#
The dashboard provides detailed information about the state of the cluster,including the running jobs, actors, workers, nodes, etc.By default, the cluster launcher and KubeRay operator will launch the dashboard, but willnot publicly expose the port.
You can securely port-forward local traffic to the dashboard via the raydashboard command.
$ ray dashboard [-p <port, 8265 by default>] <cluster config file>
The dashboard is now be visible at http://localhost:8265.
The KubeRay operator makes the dashboard available via a Service targetingthe Ray head pod, named <RayCluster name>-head-svc. You can access thedashboard from within the Kubernetes cluster at http://<RayCluster name>-head-svc:8265.
You can also view the dashboard from outside the Kubernetes cluster byusing port-forwarding:
$ kubectl port-forward service/raycluster-autoscaler-head-svc 8265:8265For more information about configuring network access to a Ray cluster onKubernetes, see the networking notes.
Changing Dashboard Ports#
CLI
To customize the port on which the dashboard runs, you can passthe --dashboard-port argument with ray start in the command line.
ray.init
If you need to customize the port on which the dashboard runs, you can pass thekeyword argument dashboard_port in your call to ray.init().
To disable the dashboard while using the “VM cluster launcher”, include the “ray start –head –include-dashboard=False” argumentand specify the desired port number in the “head_start_ray_commands” section of the cluster launcher’s YAML file.
See the Specifying non-default ports page.
Running Behind a Reverse Proxy#
The dashboard should work out-of-the-box when accessed via a reverse proxy. API requests don’t need to be proxied individually.
Always access the dashboard with a trailing / at the end of the URL.For example, if your proxy is set up to handle requests to /ray/dashboard, view the dashboard at www.my-website.com/ray/dashboard/.
The dashboard now sends HTTP requests with relative URL paths. Browsers will handle these requests as expected when the window.location.href ends in a trailing /.
This is a peculiarity of how many browsers handle requests with relative URLs, despite what MDNdefines as the expected behavior.
Make your dashboard visible without a trailing / by including a rule in your reverse proxy thatredirects the user’s browser to /, i.e. /ray/dashboard –> /ray/dashboard/.
Below is an example with a traefik TOML file that accomplishes this:
[http] [http.routers] [http.routers.to-dashboard] rule = "PathPrefix(`/ray/dashboard`)" middlewares = ["test-redirectregex", "strip"] service = "dashboard" [http.middlewares] [http.middlewares.test-redirectregex.redirectRegex] regex = "^(.*)/ray/dashboard$" replacement = "${1}/ray/dashboard/" [http.middlewares.strip.stripPrefix] prefixes = ["/ray/dashboard"] [http.services] [http.services.dashboard.loadBalancer] [[http.services.dashboard.loadBalancer.servers]] url = "http://localhost:8265"
Viewing Built-in Dashboard API Metrics#
The dashboard is powered by a server that serves both the UI code and the data about the cluster via API endpoints.There are basic Prometheus metrics that are emitted for each of these API endpoints:
ray_dashboard_api_requests_count_requests_total: Collects the total count of requests. This is tagged by endpoint, method, and http_status.
ray_dashboard_api_requests_duration_seconds_bucket: Collects the duration of requests. This is tagged by endpoint and method.
For example, you can view the p95 duration of all requests with this query:
histogram_quantile(0.95, sum(rate(ray_dashboard_api_requests_duration_seconds_bucket[5m])) by (le))
These metrics can be queried via Prometheus or Grafana UI. Instructions on how to set these tools up can be found here.
Disabling the Dashboard#
Dashboard is included in the ray[default] installation by default and automatically started.
To disable the dashboard, use the following arguments --include-dashboard.
CLI
ray start --include-dashboard=Falseray.init
import rayray.init(include_dashboard=False)
To disable the dashboard while using the “VM cluster launcher”, include the “ray start –head –include-dashboard=False” argumentin the “head_start_ray_commands” section of the cluster launcher’s YAML file.
TODO
Integrating with Prometheus and Grafana#
Setting up Prometheus#
Tip
The below instructions for Prometheus to enable a basic workflow of running and accessing the dashboard on your local machine.For more information about how to run Prometheus on a remote cluster, see here.
Ray exposes its metrics in Prometheus format. This allows us to easily scrape them using Prometheus.
First, download Prometheus. Make sure to download the correct binary for your operating system. (Ex: darwin for mac osx)
Then, unzip the archive into a local directory using the following command.
tar xvfz prometheus-*.tar.gzcd prometheus-*Ray exports metrics only when ray[default] is installed.
pip install "ray[default]"Ray provides a prometheus config that works out of the box. After running ray, it can be found at /tmp/ray/session_latest/metrics/prometheus/prometheus.yml.
global: scrape_interval: 15s evaluation_interval: 15sscrape_configs:# Scrape from each ray node as defined in the service_discovery.json provided by ray.- job_name: 'ray' file_sd_configs: - files: - '/tmp/ray/prom_metrics_service_discovery.json'
Next, let’s start Prometheus.
./prometheus --config.file=/tmp/ray/session_latest/metrics/prometheus/prometheus.ymlNote
If you are using mac, you may receive an error at this point about trying to launch an application where the developer has not been verified. See this link to fix the issue.
Now, you can access Ray metrics from the default Prometheus url, http://localhost:9090.
See here for more information on how to set up Prometheus on a Ray Cluster.
Setting up Grafana#
Tip
The below instructions for Grafana setup to enable a basic workflow of running and accessing the dashboard on your local machine.For more information about how to run Grafana on a remote cluster, see here.
Grafana is a tool that supports more advanced visualizations of prometheus metrics andallows you to create custom dashboards with your favorite metrics. Ray exports some defaultconfigurations which includes a default dashboard showing some of the most valuable metricsfor debugging ray applications.
Deploying Grafana#
First, download Grafana. Follow the instructions on the download page to download the right binary for your operating system.
Then go to to the location of the binary and run grafana using the built in configuration found in /tmp/ray/session_latest/metrics/grafana folder.
./bin/grafana-server --config /tmp/ray/session_latest/metrics/grafana/grafana.ini web
Now, you can access grafana using the default grafana url, http://localhost:3000.You can then see the default dashboard by going to dashboards -> manage -> Ray -> Default Dashboard. The same metric graphs are also accessible via Ray Dashboard.
Tip
If this is your first time using Grafana, you can login with the username: admin and password admin.
See here for more information on how to set up Grafana on a Ray Cluster.
Customizing the Prometheus export port#
Ray by default provides the service discovery file, but you can directly scrape metrics from prometheus ports.To do that, you may want to customize the port that metrics gets exposed to a pre-defined port.
ray start --head --metrics-export-port=8080 # Assign metrics export port on a head node.
Now, you can scrape Ray’s metrics using Prometheus via <ip>:8080.
Alternate Prometheus host location#
You can choose to run Prometheus on a non-default port or on a different machine. When doing so, you shouldmake sure that prometheus can scrape the metrics from your ray nodes following instructions here.
In addition, both Ray and Grafana needs to know how to access this prometheus instance. This can be configuredby setting the RAY_PROMETHEUS_HOST env var when launching ray. The env var takes in the address to access Prometheus. Moreinfo can be found here. By default, we assume Prometheus is hosted at localhost:9090.
For example, if Prometheus is hosted at port 9000 on a node with ip 55.66.77.88, One should set the value toRAY_PROMETHEUS_HOST=http://55.66.77.88:9000.
Alternate Grafana host location#
You can choose to run Grafana on a non-default port or on a different machine. If you choose to do this, theDashboard needs to be configured with a public address to that service so the web pagecan load the graphs. This can be done with the RAY_GRAFANA_HOST env var when launching ray. The env var takesin the address to access Grafana. More info can be found here. Instructionsto use an existing Grafana instance can be found here.
For the Grafana charts to work on the Ray dashboard, the user of the dashboard’s browser must be able to reachthe Grafana service. If this browser cannot reach Grafana the same way the Ray head node can, you can use a separateenv var RAY_GRAFANA_IFRAME_HOST to customize the host the browser users to attempt to reach Grafana. If this is not set,we use the value of RAY_GRAFANA_HOST by default.
For example, if Grafana is hosted at is 55.66.77.88 on port 3000. One should set the valueto RAY_GRAFANA_HOST=http://55.66.77.88:3000.
Troubleshooting#
Getting Prometheus and Grafana to use the Ray configurations when installed via homebrew on macOS X#
With homebrew, Prometheus and Grafana are installed as services that are automatically launched for you.Therefore, to configure these services, you cannot simply pass in the config files as command line arguments.
Instead, follow these instructions:1. Change the –config-file line in /usr/local/etc/prometheus.args to read --config.file /tmp/ray/session_latest/metrics/prometheus/prometheus.yml.2. Update /usr/local/etc/grafana/grafana.ini file so that it matches the contents of /tmp/ray/session_latest/metrics/grafana/grafana.ini.
You can then start or restart the services with brew services start grafana and brew services start prometheus.
MacOS does not trust the developer to install Prometheus or Grafana#
You may have received an error that looks like this:
When downloading binaries from the internet, Mac requires that the binary be signed by a trusted developer ID.Unfortunately, many developers today are not trusted by Mac and so this requirement must be overridden by the user manaully.
See these instructions on how to override the restriction and install or run the application.
Grafana dashboards are not embedded in the Ray dashboard#
If you’re getting an error that says RAY_GRAFANA_HOST is not setup despite having set it up, check that:You’ve included the protocol in the URL (e.g., http://your-grafana-url.com instead of your-grafana-url.com).The URL doesn’t have a trailing slash (e.g., http://your-grafana-url.com instead of http://your-grafana-url.com/).
Certificate Authority (CA error)#
You may see a CA error if your Grafana instance is hosted behind HTTPS. Contact the Grafana service owner to properly enable HTTPS traffic.