zoo.ray package

Submodules

zoo.ray.process module

class zoo.ray.process.ProcessInfo(out, err, errorcode, pgid, tag='default', pids=None, node_ip=None)

Bases: object

class zoo.ray.process.ProcessMonitor(process_infos, sc, ray_rdd, raycontext, verbose=False)

Bases: object

clean_fn()

print_ray_remote_err_out()

static register_shutdown_hook(pgid=None, extra_close_fn=None)

zoo.ray.process.pids_from_gpid(gpid)

zoo.ray.process.session_execute(command, env=None, tag=None, fail_fast=False, timeout=120)

zoo.ray.raycontext module

class zoo.ray.raycontext.JVMGuard

Bases: object

The registered pids would be put into the killing list of Spark Executor.

static register_pids(pids)

class zoo.ray.raycontext.RayContext(sc, redis_port=None, password='123456', object_store_memory=None, verbose=False, env=None, extra_params=None, num_ray_nodes=None, ray_node_cpu_cores=None)

Bases: object

address_info

classmethod get(initialize=True)

init(driver_cores=0)

Initiate the ray cluster.

Parameters:driver_cores – The number of cores for the raylet on driver for Spark cluster mode.

Default is 0 and in this case the local driver wouldn’t have any ray workload.

:return The dictionary of address information about the ray cluster. Information contains node_ip_address, redis_address, object_store_address, raylet_socket_name, webui_url and session_dir.

purge()

Invoke ray stop to clean ray processes.

stop()

class zoo.ray.raycontext.RayServiceFuncGenerator(python_loc, redis_port, ray_node_cpu_cores, password, object_store_memory, verbose=False, env=None, extra_params=None)

Bases: object

This should be a pickable class.

gen_ray_start(master_ip)

gen_stop()

zoo.ray.raycontext.kill_redundant_log_monitors(redis_address)

Killing redundant log_monitor.py processes. If multiple ray nodes are started on the same machine, there will be multiple ray log_monitor.py processes monitoring the same log dir. As a result, the logs will be replicated multiple times and forwarded to driver. See issue https://github.com/ray-project/ray/issues/10392

zoo.ray.utils module

zoo.ray.utils.gen_shutdown_per_node(pgids, node_ips=None)

zoo.ray.utils.is_local(sc)

zoo.ray.utils.resource_to_bytes(resource_str)

zoo.ray.utils.to_list(input)

Module contents