408 lines
18 KiB
HTML
Executable file
408 lines
18 KiB
HTML
Executable file
<?xml version="1.0" encoding="ascii"?>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
|
|
"DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
|
|
<head>
|
|
<title>Scientific.DistributedComputing.MasterSlave</title>
|
|
<link rel="stylesheet" href="epydoc.css" type="text/css" />
|
|
<script type="text/javascript" src="epydoc.js"></script>
|
|
</head>
|
|
|
|
<body bgcolor="white" text="black" link="blue" vlink="#204080"
|
|
alink="#204080">
|
|
<!-- ==================== NAVIGATION BAR ==================== -->
|
|
<table class="navbar" border="0" width="100%" cellpadding="0"
|
|
bgcolor="#a0c0ff" cellspacing="0">
|
|
<tr valign="middle">
|
|
<!-- Home link -->
|
|
<th> <a
|
|
href="Scientific-module.html">Home</a> </th>
|
|
|
|
<!-- Tree link -->
|
|
<th> <a
|
|
href="module-tree.html">Trees</a> </th>
|
|
|
|
<!-- Index link -->
|
|
<th> <a
|
|
href="identifier-index.html">Indices</a> </th>
|
|
|
|
<!-- Help link -->
|
|
<th> <a
|
|
href="help.html">Help</a> </th>
|
|
|
|
<!-- Project homepage -->
|
|
<th class="navbar" align="right" width="100%">
|
|
<table border="0" cellpadding="0" cellspacing="0">
|
|
<tr><th class="navbar" align="center"
|
|
><a class="navbar" target="_top" href="http://dirac.cnrs-orleans.fr/ScientificPython/">Scientific Python</a></th>
|
|
</tr></table></th>
|
|
</tr>
|
|
</table>
|
|
<table width="100%" cellpadding="0" cellspacing="0">
|
|
<tr valign="top">
|
|
<td width="100%">
|
|
<span class="breadcrumbs">
|
|
<a href="Scientific-module.html">Package Scientific</a> ::
|
|
<a href="Scientific.DistributedComputing-module.html">Package DistributedComputing</a> ::
|
|
Module MasterSlave
|
|
</span>
|
|
</td>
|
|
<td>
|
|
<table cellpadding="0" cellspacing="0">
|
|
<!-- hide/show private -->
|
|
<tr><td align="right"><span class="options"
|
|
>[<a href="frames.html" target="_top">frames</a
|
|
>] | <a href="Scientific.DistributedComputing.MasterSlave-module.html"
|
|
target="_top">no frames</a>]</span></td></tr>
|
|
</table>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<!-- ==================== MODULE DESCRIPTION ==================== -->
|
|
<h1 class="epydoc">Module MasterSlave</h1><p class="nomargin-top"></p>
|
|
<p>Distributed computing using a master-slave model</p>
|
|
<p>The classes in this module provide a simple way to parallelize
|
|
independent computations in a program. The communication is handled by
|
|
the Pyro package, which must be installed before this module can be used.
|
|
Pyro can be obtained from http://pyro.sourceforge.net/. By default, the
|
|
Pyro name server is used to initialize communication. Please read the
|
|
Pyro documentation for learning how to use the name server.</p>
|
|
<p>The principle of the master-slave model is that there is a single
|
|
master process that defines computational tasks and any number of slave
|
|
processes that execute these tasks. The master defines task requests and
|
|
then waits for the results to come in. The slaves wait for a task
|
|
request, execute it, return the result, and wait for the next task. There
|
|
can be any number of slave processes, which can be started and terminated
|
|
independently, the only condition being that no slave process can be
|
|
started before its master process. This setup makes it possible to
|
|
perform a lengthy computation using a variable number of processors.</p>
|
|
<p>Communication between the master and the slave processes passes
|
|
through a TaskManager object that is created automatically as part of the
|
|
master process. The task manager stores and hands out task requests and
|
|
results. The task manager also keeps track of the slave processes. When a
|
|
slave process disappears (because it was killed or because of a hardware
|
|
failure), the task manager re-schedules its active task(s) to another
|
|
slave process. This makes the master-slave system very fault
|
|
tolerant.</p>
|
|
<p>Each task manager has a label that makes it possible to distinguish
|
|
between several master-slave groups running at the same time. It is by
|
|
the label that slave processes identify the master process for which they
|
|
work.</p>
|
|
<p>The script "task_manager" prints statistics about a
|
|
currently active task manager; it takes the label as an argument. It
|
|
shows the number of currently active processes (master plus slaves), the
|
|
number of waiting and running tasks, and the number of results waiting to
|
|
be picked up.</p>
|
|
<p>The script Examples/master_slave_demo.py illustrates the use of the
|
|
master-slave setup in a simple script. Both master and slave processes
|
|
are defined in the same script. The scripts Examples/master.py and
|
|
Examples/slave.py show a master-slave setup using two distinct scripts.
|
|
This is more flexible because task requests and result retrievals can be
|
|
made from anywhere in the master code.</p>
|
|
|
|
<!-- ==================== CLASSES ==================== -->
|
|
<a name="section-Classes"></a>
|
|
<table class="summary" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr bgcolor="#70b0f0" class="table-header">
|
|
<td align="left" colspan="2" class="table-header">
|
|
<span class="table-header">Classes</span></td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<a href="Scientific.DistributedComputing.MasterSlave.GlobalStateValue-class.html" class="summary-name">GlobalStateValue</a>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<a href="Scientific.DistributedComputing.MasterSlave.MasterProcess-class.html" class="summary-name">MasterProcess</a><br />
|
|
Master process in a master-slave setup
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<a href="Scientific.DistributedComputing.MasterSlave.SlaveProcess-class.html" class="summary-name">SlaveProcess</a><br />
|
|
Slave process in a master-slave setup
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<!-- ==================== FUNCTIONS ==================== -->
|
|
<a name="section-Functions"></a>
|
|
<table class="summary" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr bgcolor="#70b0f0" class="table-header">
|
|
<td align="left" colspan="2" class="table-header">
|
|
<span class="table-header">Functions</span></td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr>
|
|
<td><span class="summary-sig"><a name="getMachineInfo"></a><span class="summary-sig-name">getMachineInfo</span>()</span></td>
|
|
<td align="right" valign="top">
|
|
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"><a
|
|
href="Scientific.DistributedComputing.MasterSlave.MasterProcess-class.html"
|
|
class="link">MasterProcess</a></span>
|
|
</td><td class="summary">
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr>
|
|
<td><span class="summary-sig"><a href="Scientific.DistributedComputing.MasterSlave-module.html#initializeMasterProcess" class="summary-sig-name">initializeMasterProcess</a>(<span class="summary-sig-arg">label</span>,
|
|
<span class="summary-sig-arg">slave_script</span>=<span class="summary-sig-default">None</span>,
|
|
<span class="summary-sig-arg">slave_module</span>=<span class="summary-sig-default">None</span>,
|
|
<span class="summary-sig-arg">use_name_server</span>=<span class="summary-sig-default">True</span>)</span><br />
|
|
Initializes a master process.</td>
|
|
<td align="right" valign="top">
|
|
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr>
|
|
<td><span class="summary-sig"><a href="Scientific.DistributedComputing.MasterSlave-module.html#runJob" class="summary-sig-name">runJob</a>(<span class="summary-sig-arg">label</span>,
|
|
<span class="summary-sig-arg">master_class</span>,
|
|
<span class="summary-sig-arg">slave_class</span>,
|
|
<span class="summary-sig-arg">watchdog_period</span>=<span class="summary-sig-default">120.0</span>,
|
|
<span class="summary-sig-arg">launch_slaves</span>=<span class="summary-sig-default">0</span>)</span><br />
|
|
Creates an instance of the master_class and runs it.</td>
|
|
<td align="right" valign="top">
|
|
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr>
|
|
<td><span class="summary-sig"><a href="Scientific.DistributedComputing.MasterSlave-module.html#startSlaveProcess" class="summary-sig-name">startSlaveProcess</a>(<span class="summary-sig-arg">label</span>=<span class="summary-sig-default">None</span>,
|
|
<span class="summary-sig-arg">master_host</span>=<span class="summary-sig-default">None</span>)</span><br />
|
|
Starts a slave process.</td>
|
|
<td align="right" valign="top">
|
|
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<!-- ==================== VARIABLES ==================== -->
|
|
<a name="section-Variables"></a>
|
|
<table class="summary" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr bgcolor="#70b0f0" class="table-header">
|
|
<td align="left" colspan="2" class="table-header">
|
|
<span class="table-header">Variables</span></td>
|
|
</tr>
|
|
<tr>
|
|
<td width="15%" align="right" valign="top" class="summary">
|
|
<span class="summary-type"> </span>
|
|
</td><td class="summary">
|
|
<a name="debug"></a><span class="summary-name">debug</span> = <code title="False">False</code>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
<!-- ==================== FUNCTION DETAILS ==================== -->
|
|
<a name="section-FunctionDetails"></a>
|
|
<table class="details" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr bgcolor="#70b0f0" class="table-header">
|
|
<td align="left" colspan="2" class="table-header">
|
|
<span class="table-header">Function Details</span></td>
|
|
</tr>
|
|
</table>
|
|
<a name="initializeMasterProcess"></a>
|
|
<div>
|
|
<table class="details" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr><td>
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr valign="top"><td>
|
|
<h3 class="epydoc"><span class="sig"><span class="sig-name">initializeMasterProcess</span>(<span class="sig-arg">label</span>,
|
|
<span class="sig-arg">slave_script</span>=<span class="sig-default">None</span>,
|
|
<span class="sig-arg">slave_module</span>=<span class="sig-default">None</span>,
|
|
<span class="sig-arg">use_name_server</span>=<span class="sig-default">True</span>)</span>
|
|
</h3>
|
|
</td><td align="right" valign="top"
|
|
>
|
|
</td>
|
|
</tr></table>
|
|
|
|
<p>Initializes a master process.</p>
|
|
<dl class="fields">
|
|
<dt>Parameters:</dt>
|
|
<dd><ul class="nomargin-top">
|
|
<li><strong class="pname"><code>label</code></strong> (<code>str</code>) - the label that identifies the task manager</li>
|
|
<li><strong class="pname"><code>slave_script</code></strong> (<code>str</code>) - the file name of the script that defines the corresponding slave
|
|
process</li>
|
|
<li><strong class="pname"><code>slave_module</code></strong> (<code>str</code>) - the name of the module that defines the corresponding slave
|
|
process</li>
|
|
<li><strong class="pname"><code>use_name_server</code></strong> (<code>bool</code>) - If <code>True</code> (default), the task manager is registered
|
|
with the Pyro name server. If <code>False</code>, the name server
|
|
is not used and slave processes need to know the host on which
|
|
the master process is running.</li>
|
|
</ul></dd>
|
|
<dt>Returns: <a
|
|
href="Scientific.DistributedComputing.MasterSlave.MasterProcess-class.html"
|
|
class="link">MasterProcess</a></dt>
|
|
<dd>a process object on which the methods requestTask() and
|
|
retrieveResult() can be called.</dd>
|
|
</dl>
|
|
</td></tr></table>
|
|
</div>
|
|
<a name="runJob"></a>
|
|
<div>
|
|
<table class="details" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr><td>
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr valign="top"><td>
|
|
<h3 class="epydoc"><span class="sig"><span class="sig-name">runJob</span>(<span class="sig-arg">label</span>,
|
|
<span class="sig-arg">master_class</span>,
|
|
<span class="sig-arg">slave_class</span>,
|
|
<span class="sig-arg">watchdog_period</span>=<span class="sig-default">120.0</span>,
|
|
<span class="sig-arg">launch_slaves</span>=<span class="sig-default">0</span>)</span>
|
|
</h3>
|
|
</td><td align="right" valign="top"
|
|
>
|
|
</td>
|
|
</tr></table>
|
|
|
|
<p>Creates an instance of the master_class and runs it. A copy of the
|
|
script and the current working directory are stored in the TaskManager
|
|
object to enable the task_manager script to launch slave processes.</p>
|
|
<dl class="fields">
|
|
<dt>Parameters:</dt>
|
|
<dd><ul class="nomargin-top">
|
|
<li><strong class="pname"><code>label</code></strong> (<code>str</code>) - the label that identifies the task manager</li>
|
|
<li><strong class="pname"><code>master_class</code></strong> - the class implementing the master process (a subclass of <a
|
|
href="Scientific.DistributedComputing.MasterSlave.MasterProcess-class.html"
|
|
class="link">MasterProcess</a>)</li>
|
|
<li><strong class="pname"><code>slave_class</code></strong> - the class implementing the slave process (a subclass of <a
|
|
href="Scientific.DistributedComputing.MasterSlave.SlaveProcess-class.html"
|
|
class="link">SlaveProcess</a>)</li>
|
|
<li><strong class="pname"><code>watchdog_period</code></strong> (<code>int</code> or <code>NoneType</code>) - the interval (in seconds) at which the slave process sends
|
|
messages to the manager to signal that it is still alive. If
|
|
None, no messages are sent at all. In that case, the manager
|
|
cannot recognize if the slave job has crashed or been killed.</li>
|
|
<li><strong class="pname"><code>launch_slaves</code></strong> (<code>int</code>) - the number of slaves jobs to launch immediately on the same
|
|
machine that runs the master process</li>
|
|
</ul></dd>
|
|
</dl>
|
|
</td></tr></table>
|
|
</div>
|
|
<a name="startSlaveProcess"></a>
|
|
<div>
|
|
<table class="details" border="1" cellpadding="3"
|
|
cellspacing="0" width="100%" bgcolor="white">
|
|
<tr><td>
|
|
<table width="100%" cellpadding="0" cellspacing="0" border="0">
|
|
<tr valign="top"><td>
|
|
<h3 class="epydoc"><span class="sig"><span class="sig-name">startSlaveProcess</span>(<span class="sig-arg">label</span>=<span class="sig-default">None</span>,
|
|
<span class="sig-arg">master_host</span>=<span class="sig-default">None</span>)</span>
|
|
</h3>
|
|
</td><td align="right" valign="top"
|
|
>
|
|
</td>
|
|
</tr></table>
|
|
|
|
<p>Starts a slave process. Must be called at the end of a script that
|
|
defines or imports all task handlers.</p>
|
|
<dl class="fields">
|
|
<dt>Parameters:</dt>
|
|
<dd><ul class="nomargin-top">
|
|
<li><strong class="pname"><code>label</code></strong> (<code>str</code> or <code>NoneType</code>) - the label that identifies the task manager. May be omitted if the
|
|
slave process is started through the task_manager script.</li>
|
|
<li><strong class="pname"><code>master_host</code></strong> (<code>str</code> or <code>NoneType</code>) - If <code>None</code> (default), the task manager of the master
|
|
process is located using the Pyro name server. If no name server
|
|
is used, this parameter must be the hostname of the machine on
|
|
which the master process runs, plus the port number if it is
|
|
different from the default (7766).</li>
|
|
</ul></dd>
|
|
</dl>
|
|
</td></tr></table>
|
|
</div>
|
|
<br />
|
|
<!-- ==================== NAVIGATION BAR ==================== -->
|
|
<table class="navbar" border="0" width="100%" cellpadding="0"
|
|
bgcolor="#a0c0ff" cellspacing="0">
|
|
<tr valign="middle">
|
|
<!-- Home link -->
|
|
<th> <a
|
|
href="Scientific-module.html">Home</a> </th>
|
|
|
|
<!-- Tree link -->
|
|
<th> <a
|
|
href="module-tree.html">Trees</a> </th>
|
|
|
|
<!-- Index link -->
|
|
<th> <a
|
|
href="identifier-index.html">Indices</a> </th>
|
|
|
|
<!-- Help link -->
|
|
<th> <a
|
|
href="help.html">Help</a> </th>
|
|
|
|
<!-- Project homepage -->
|
|
<th class="navbar" align="right" width="100%">
|
|
<table border="0" cellpadding="0" cellspacing="0">
|
|
<tr><th class="navbar" align="center"
|
|
><a class="navbar" target="_top" href="http://dirac.cnrs-orleans.fr/ScientificPython/">Scientific Python</a></th>
|
|
</tr></table></th>
|
|
</tr>
|
|
</table>
|
|
<table border="0" cellpadding="0" cellspacing="0" width="100%%">
|
|
<tr>
|
|
<td align="left" class="footer">
|
|
Generated by Epydoc 3.0 on Tue Oct 28 14:15:59 2008
|
|
</td>
|
|
<td align="right" class="footer">
|
|
<a target="mainFrame" href="http://epydoc.sourceforge.net"
|
|
>http://epydoc.sourceforge.net</a>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
<script type="text/javascript">
|
|
<!--
|
|
// Private objects are initially displayed (because if
|
|
// javascript is turned off then we want them to be
|
|
// visible); but by default, we want to hide them. So hide
|
|
// them unless we have a cookie that says to show them.
|
|
checkCookie();
|
|
// -->
|
|
</script>
|
|
</body>
|
|
</html>
|