TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013
END-USER GUIDE

TECHILA WITH PYTHON
JANUARY 04, 2013
TECHILA WITH PYTHON - JANUARY 04, 2013
Disclaimer

Techila Technologies Ltd. disclaims any and all warranties, express, implied or statutory regarding this
document or the use of thereof by you to the full extent permitted by law. Without limiting the generality of
the foregoing, this document provided by Techila Technologies Ltd. in connection therewith are provided
'as-is' and without warranties of any kind, including, without limitation, any warranties of performance or
implied warranties of merchantability, fitness for a particular purpose, title and noninfringement. Further,
Techila Technologies Ltd. does not make, and has not made, any presentation or warranty that the
document is accurate, complete, reliable, current, error-free, or free from harmful information.

Limitation of Liability
In no event shall Techila Technologies Ltd. or any of its respective directors, officers, employees, or
agents, be liable to you or any other person or entity, under any theory, including without limitation
negligence, for damages of any kind arising from or related to the application of this document or any
information, content, or materials in or accessible through this document, including, but not limited to,
direct, indirect, actual, incidental, punitive, special or consequential damages, lost income, revenue or
profits, lost or damaged data, or other commercial or economic loss, that result from your use of, or
inability to use, this document, even if any of those persons or entities have been advised of the possibility
of such damages or such damages are foreseeable.

Use of this document and copyright
No part of this document may be used, reproduced, modified, or transmitted in any form or means without
the prior written permission of Techila Technologies. This document and the product it describes are
considered protected by copyrights and other intellectual property rights according to the applicable laws.

Copyright Techila Technologies 2013. All rights reserved.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
TECHILA WITH PYTHON - JANUARY 04, 2013
Table of contents
1   Introduction.............................................................................................................................................. 4
  1.1      Installing the techila package ........................................................................................................ 5
  1.2      Example material .......................................................................................................................... 5
  1.3      Naming convention of the python scripts ...................................................................................... 6
  1.4      Python peach-function .................................................................................................................. 6
  1.5      Process flow in computational Projects ........................................................................................ 7
        1.5.1          The peach-function ........................................................................................................ 7
  1.6      Techila environment variables ...................................................................................................... 8
2   Peach tutorial examples .......................................................................................................................... 9
  2.1      Executing a simple Python function on Workers ........................................................................ 10
  2.2      Using input parameters ............................................................................................................... 14
  2.3      Transferring data files ................................................................................................................. 19
  2.4      Multiple functions in an Python script ......................................................................................... 23
3   Peach feature examples ....................................................................................................................... 26
  3.1      Monte Carlo Pi with peach .......................................................................................................... 28
  3.2      Streaming & Callback Function................................................................................................... 31
  3.3      Job Input Files ............................................................................................................................. 35
  3.4      Project Detaching ........................................................................................................................ 38
  3.5      Iterative Projects ......................................................................................................................... 41
  3.6      Data Bundles............................................................................................................................... 44
  3.7      Function handle .......................................................................................................................... 47
  3.8      File Handler ................................................................................................................................. 49
  3.9      Snapshots ................................................................................................................................... 52
  3.10     Using custom Python packages in computational Projects ........................................................ 55
  3.11     Precompiled binaries .................................................................................................................. 61




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
TECHILA WITH PYTHON - JANUARY 04, 2013
End-User Guide                      Techila with Python                                 4/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013




1     Introduction
This document is intended for Techila End-Users who are using Python as their main development
environment. If you are unfamiliar with Techila terminology or the operating principles of the Techila
system, please see document "Techila Fundamentals" for more information.

The structure of this document is as follows:

Chapter 1 contains important information regarding the installation of required Python packages that
enable you to use Techila with Python. This Chapter also contains a brief introduction on the naming
convention of the Python-scripts and introduces the peach-function, which is used for distributing
computations from Python to the Techila environment.

Chapter 2 contains walkthroughs of simplistic example code samples that use the peach-function. The
example material illustrates how to control the core features of the peach-function, including defining input
arguments for the executable function, transferring data files to the Workers and calling different functions
from the Python script that is evaluated on the Worker. After examining the material in this Chapter you
should be able split a simple locally executable program into two pieces of code (Local Control Code and
Worker Code), which in turn can be used to perform the computations in the Techila environment.

Chapter 3 contains walkthroughs of several examples that illustrate how to implement different features
available in the peach-function. Each subchapter in this Chapter contains a walkthrough of an executable
piece of code that illustrates how to implement one or more peach-function features. Each Chapter is
named according to the feature that will be the focused on. After examining the material in this Chapter
you should be able implement several features available in the peach-function in your own distributed
application.

Please note that screenshots in this document are from a Windows 7 operating system.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
TECHILA WITH PYTHON - JANUARY 04, 2013
End-User Guide                      Techila with Python                                 5/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


1.1     Installing the techila package
The 'techila' Python package is included in the Techila SDK and contains all Techila related Python
functions required for creating computational Projects.

Please follow the steps below to install the package.

      1. Launch a command prompt \ terminal

      2. Change your current working directory to the following directory:

          cd \techila\lib\python

      3. Install the 'techila' package using command:

          python setup.py install




         Figure 1.Installing the 'techila' package.


1.2     Example material
The Python scripts containing the example material discussed in this document can be found in the
Tutorial and Features folders in the Techila SDK. These folders contain subfolders, which contain the
actual Python scripts that can be used to run the examples.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
TECHILA WITH PYTHON - JANUARY 04, 2013
End-User Guide                      Techila with Python                                 6/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013




                                  techila


                                                 examples


                                                             python


                                                                          Tutorial                      Chapter 2

                                                                          Features                      Chapter 3

Figure 2. The example material discussed in this document can be found in the in the 'python'
           folder in the Techila SDK

1.3       Naming convention of the python scripts
The typical naming convention of Python scripts presented in this document is explained below:

          Python scripts ending with '_dist' contain the Worker Code, which will be distributed to the
           Workers when the Local Control Code is executed.
          Python scripts beginning with 'run_' contain the Local Control Code, which will create the
           computational Project when executed locally on the End-User’s own computer.
          Python scripts beginning with 'local_' contain locally executable code, which does not
           communicate with the Techila environment.

Please note that some Python scripts and functions might be named differently, depending on their role in
the computational Project.

1.4       Python peach-function
The peach-function provides a simple interface that can be used to distribute Python programs or
precompiled binaries. Each peach-function input argument is a named parameter, which refer to a
computer language's support for function calls that state the name of each parameter within the function
call itself.

A minimalistic Python peach-function syntax typically includes the following parameters:

          funcname
          params
          files
          peachvector
          datafiles

These parameters can be used to define input arguments for the executable function and transfer
additional files to the Workers. An example of a peach-function syntax using these parameters is shown
below:


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                 7/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013



techila.peach(funcname = 'example_function',                                #   Function executed on Workers
              params = [var1,var2],                                         #   Input arguments to the function
              files = ['codefile.py'],                                      #   Evaluated at the start of a Job
              datafiles = ['file1','file2'],                                #   Files transferred to Workers
              peachvector = [2,4,6,8,10]                                    #   Peachvector definition
              )

Examples and more detailed explanations of these parameters can be found in Chapter 2. General
information on available peach-function parameters can also be displayed by executing the following
commands in Python.

import techila
help(techila.peach)

1.5        Process flow in computational Projects
When a Project is created with the peach-function, each Job in a computational Project will have a
separate Python session. When a Job is started on a Worker, functions and variables are loaded by
evaluating the Python script that was defined in the 'files' parameter and by loading the parameters
stored in the 'techila_peach_inputdata'file. These parameters will include the parameters defined
in the 'params' peach-function parameter.

When a Job is started on a Worker, the 'peachclient.py' script (included in the 'techila'
package) is called. The 'peachclient.py' file is a Python script that acts as a wrapper for the Worker
Code and is responsible for calling the executable function, passing input arguments to the function and
returning the final computational results. This functionality is hidden from the End-User. The
'peachclient.py' will be used automatically in computational Projects created with peach-function.

The 'peachclient.py' wrapper also sets a preliminary seed for the random number generator by using
the Python seed() function from the package 'random'. Each Job in a computational Project will receive
a unique random number seed based on the current system time and the 'jobidx' parameter, which will
be different for each Job. The preliminary random number seeding can be overridden by calling the
seed() function in the Worker Code with the desired random seed.

1.5.1 The peach-function
The list below contains some of the Python specific activities that are performed automatically when the
peach-function is used to create a computational Project.

      1.  The peach-function is called locally on the End-Users computer
      2.  Python scripts listed in the 'files' parameter are transferred to Workers
      3.  Files listed in the 'datafiles' parameter are transferred to Workers
      4.  The 'peachclient.py' file is transferred to Workers
      5.  Input parameters listed in the 'params' parameter are stored in a file called
          'techila_peach_inputdata', which is transferred to Workers.
      6. The files listed in the 'files' and 'datafiles' parameters and the files
          'techila_peach_inputdata'and 'peachclient.py' are copied to the temporary working
          directory on the Worker
      7. The 'peachclient.py' wrapper is called on the Worker.
      8. Variables stored in the file 'techila_peach_inputdata' are loaded
      9. Files listed in the 'files' parameter are evaluated using the Python 'execfile' command
      10. The '' notation is replaced with a 'peachvector' element


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                 8/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

      11. The peachclient calls the function defined in the 'funcname' parameter with the input parameters
          defined in 'params'.
      12. The peachclient saves the result in to a file, which is returned from the Worker to the End-User
      13. The peach-function reads the output file and stores the result in a list element (If a callback function is
          used, the result of the callback function is returned).
      14. The entire list is returned by the peach-function.

1.6     Techila environment variables
The table below contains descriptions of environment variables used by Techila. These environment
variables can be used to define e.g. the path of the 'techila' directory on your computer or to define the
version of Python runtime components used during computational Projects.

 Environment variable                       Description                                                         Example
                                                                                                                value
 TECHILA_SDKROOT                            Defines the path of the root directory of the Techila               C:\techila
                                            SDK.

 TECHILA_PYTHON_VERSION                     Defines the Python Runtime Bundle version that will                 273
                                            be used in Projects. This parameter can be used
                                            e.g. if the Techila environment does not have a
                                            Python Runtime Bundle with the same version as
                                            your local Python environment.

                                            The version of the Runtime Bundle is defined using
                                            the syntax . (As returned by
                                            command 'sys.version_info' in Python)




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                 9/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


2     Peach tutorial examples
This Chapter contains four minimalistic examples on how to implement and control the core features of the
peach-function. The example material discussed in this Chapter, including Python scripts and data files
can be found in the subdirectories under the following folder in the Techila SDK:

         techila\examples\python\Tutorial

Each of the examples contains three pieces of code:

         A script containing a locally executable Python function. This function will be executed locally and
          will not communicate with the distributed computing environment in any way. This script is
          provided as reference material to illustrate what modifications are required to execute the
          computations in the Techila environment.
         A script containing the Local Control Code, which will be executed locally on the End-Users
          computer. This script contains the peach-function call, which will distribute the computations in the
          Worker Code to the distributed computing environment
         A script containing the Worker Code, which will be executed on the Workers. This script contains
          the computationally intensive part of the locally executable script.

Please note that the example material in this Chapter is only intended to illustrate the core mechanics
related to distributing computation with the peach-function. More information on available features can be
found in Chapter 3 and by executing the following commands in Python.

import techila
help(techila.peach)




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                10/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


2.1       Executing a simple Python function on Workers
This example is intended to provide an introduction on distributed computing using Techila with Python
using the peach-function. The purpose of this example is to:

          Demonstrate how to modify a simple, locally executable Python script that contains one function
           so that the computational operations will be performed in the Techila environment
          Demonstrate the difference between Local Control Code and Worker Code in a Python
           environment
          Demonstrate the basic syntax of the peach-function in a Python environment

The material discussed in this example is located in the following folder in the Techila SDK:

          techila\examples\python\Tutorial\1_distribution

Locally executable Python function

The Python script 'local_function.py' contains one function called 'local_function', which
consists of one for-loop. The algorithm of the locally executable function used in this example is shown
below.

                              def local_function(x):
                                  result = []
                                  for j in range(x):
                                      result.append(1 + 1)
                                  return result


The function takes one input argument, which defines the number of iterations that will be performed in the
for-loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will
be appended to the 'result' list.

For example, when performing five iterations the 'result' list would contain the following values.

                                      Number of iterations: 5
                                   index        0 1 2     3   4
                                   value        2 2 2     2   2

To execute the locally executable function, use the commands shown below:

execfile('local_function.py')
result=local_function(5)

The output generated by the locally executable program should resemble the one shown below in Figure
3.




Figure 3. Executing the locally executable function in Python.
©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                11/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013



Distributed version of the program

All arithmetic operations in the locally executable function are performed in the for-loop. There are no
recursive data dependencies between iterations, meaning that the all the iterations can be performed
simultaneously. The iterations can be performed simultaneously by placing the arithmetic operations to a
separate file (distribution_dist.py), which can be then transferred and executed on Workers.

The file containing the Local Control Code ('run_distribution.py') will be used to create the
computational Project. The Worker Code ('distribution_dist.py') will be transferred to the
Workers where the file will automatically be evaluated (using 'execfile') at the preliminary stages of
the Job. After the file has been evaluated, the function 'distribution_dist' will be executed. The
result returned by the function will be returned from the Job.

Local Control Code

The Local Control Code used to create the computational Project is shown below.

           1     def run_distribution(jobcount):
           2         import techila
           3         result = techila.peach(funcname = 'distribution_dist',
           4                                files = 'distribution_dist.py',
           5                                jobs=jobcount
           6                                )
           7         print(result)
           8         return(result)

Line 1 contains the definition for the function 'run_distribution'. This function takes one input
argument called 'jobcount' (integer). This variable will be used during the peach-function call to define
the value of the 'jobs' parameter, which defines the number of Jobs in the Project.

Line 2 imports the 'techila' package, which is required to use the Techila Python functions such as
'peach'.

Lines 3 to 6 contain the peach-function call, which will be used to create the computational Project. After
the computational Project has been completed, the results will be stored in the 'result' list. The number
of list elements will be the same as the number of Jobs in the Project. Each list element will contain the
result returned from one Job.

The parameters of the peach-function call are explained below.

funcname = 'distribution_dist'

The 'funcname' parameter shown above defines that the function 'distributed_dist' will be
executed in each Job. This function will be defined when the file 'distribution_dist.py' is
evaluated as explained below.

files = 'distribution_dist.py'

The parameter shown above defines that a file named 'distributed_dist_py' should be evaluated
with the 'execfile' command at the preliminary stage of the computational Job. This evaluation will
make all variables and functions defined in the file accessible during the computational Job. In this
example, the file contains the definition for the 'distribution_dist' function.

jobs = jobcount


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                12/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013



The 'jobs' parameter shown above defines the number of Jobs in the Project. In this example, the
number of Jobs will be based on the value of the 'jobcount' variable. The number of Jobs in the Project
could also be defined by using the 'peachvector' parameter. For more information on how to define the
'peachvector', please see Chapter 2.2.

Line 7 contains a print command that will be used to print the 'result' list. Each element in the array
will contain the result for one Job in the Project.

Line 8 will return the 'result' list as the output.

Worker Code

The Worker Code that will be executed on the Workers is in the file called 'distribution_dist.py'.
The content of the file is shown below.


                                    def distribution_dist():
                                        result = 1 + 1
                                        return(result)


Operations performed in the Worker Code correspond to those performed during one iteration of the
locally executable for-loop structure. Each Job will simply sum two integers (1+1) and store the value of
the summation to the 'result' variable. The 'result' variable will returned from the Job will be
eventually received by the End-User as a one of the list elements returned by the peach-function in the
Local Control Code.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 4.




Figure 4. The names of Python scripts that will be evaluated using 'execfile' on the Worker are
defined using the 'files' parameter. In this example, the file 'distribution_dist.py' will be transferred
to all Workers and evaluated at the preliminary stages of the computational Job. The function that
will be called is defined with the 'funcname' parameter. In this example, the function
'distribution_dist' will be called in each computational Job.


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                13/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

Creating the computational Project

To create the computational Project, change your current working directory (in Python) to the directory
containing the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown
below:

execfile('run_distribution.py')
result = run_distribution(5)

If prompted, enter the password to your keystore file. The computation Project will be created and
information on the Project progress will be displayed in the console. The output generated by the
commands should resemble the one shown below in Figure 5.




Figure 5. Creating the computational Project in a Python session.

The computational Jobs will be extremely short as each Job consists of simply summing up two integers;
1+1. The computations occurring during the Project are illustrated below in Figure 6:




Figure 6. The input argument passed to the function 'run_distribution' will be used to determine the
number of Jobs in the Project. The same arithmetic operation, 1+1, is performed in each Job.
Results are delivered back to the End-Users computer where they will be stored as list elements in
the 'result' list.

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                14/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


2.2       Using input parameters
This purpose of this example is to demonstrate:

          How to transfer input parameters to the executable function

In this example, parameters will be transferred to the Workers and passed to the executable function as
input arguments. This will be performed by using the 'params' parameter of the peach-function.

The general syntax for defining input arguments for the executable function is shown below:

params = 

The '' notation will need to be replaced with a comma separated list of
variables that you wish to pass as input arguments. For example, the following syntax would define two
input arguments 'var1' and 'var2' for the function that will be executed on Workers.

params = [var1, var2]

The values of the input arguments will be read from your current Python session, meaning you can
transfer any parameters that are currently defined.

Dynamic input arguments (arguments that will have a different value for each Job) can be passed to the
executable function by using the '' notation. This notation will be replaced by a different element
of the 'peachvector' in each Job.

For example, the following syntax will pass three input arguments for the executable function. The first two
input arguments ('var1' and 'var2') have the same values for all Jobs. The third input argument
('') will be replaced with 'peachvector' elements. Job #1 will receive the first element (value
2), Job #2 will receive the second element (value 4) and so on.

params = [var1, var2, ''],
peachvector = [2,4,6,8,10]

Please note that when defining the 'peachvector', the length of the 'peachvector' will also define
the number of Jobs in the Project. With the example shown above, the Project would consist of five Jobs.

Another parameter that can be used when performing parameter sweep type computations is the
'vecidx' parameter. When used, this notation will be replaced by the corresponding 'peachvector'
index for each Job.

For example, the following syntax will pass two input arguments to the executable function. The first input
argument ('') will be replaced with 'peachvector' elements and the second input argument
('') with the corresponding 'peachvector' index values.

params = ['', ''],
peachvector = [2,4,6,8,10]

The input arguments that would be passed to the executable functions during different Jobs are shown
below.
                      Argument #1 ('')         Argument #2 ('')
           Job #1     2                               0
           Job #2     4                               1
           Job #3     6                               2
           Job #4     8                               3
           Job #5     10                              4


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                15/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013



The material discussed in this example is located in the following folder in the Techila SDK:

         techila\examples\python\Tutorial\2_parameters

Locally executable Python function

The algorithm for the locally executable function used in this example is shown below.


                           def local_function(multip, loops):
                               result = []
                               for x in range(1, loops + 1):
                                   result.append(multip * x)
                               print(result)
                               return(result)


This function takes two input arguments; 'multip' and 'loops'. The parameter 'loops' determines
the number of iterations in the for-loop. The parameter 'multip' is a number, which will be multiplied
with the iteration counter represented by 'x'. The result of this arithmetic operation will be appended to a
list called 'result', which will be returned as the output value of the function.

An example 'result' list (five iterations) is shown below.

                                              multip = 2; loops=5
                                         index       0   1  2   3                    4
                                         result      2   4  6   8                    10

The locally executable function can be executed using the commands shown below:

execfile('local_function.py')
result=local_function(2,5)

After executing the commands, numerical values stored in the 'result' list will be displayed. If you
executed the function using the values shown above, the output generated by the function should
resemble the one shown below in Figure 7




Figure 7. Executing the locally executable function in Python.

Distributed version of the program

All the computations in locally executable Python function are performed in the for-loop and there are no
dependencies between the iterations. As a result of this, the locally executable program can be converted
to a distributed version by extracting the arithmetic operation into a separate piece of code
('parameters_dist.py'), which will be executed on Workers.


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                16/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

The Local Control Code ('run_parameters.py') contains the peach-function call that will be used to
create the computational Project. In order to perform similar operations in the Project as in the local
version, input arguments need to be given to the executable function to simulate different iterations of the
for-loop. These input arguments will be transferred by the using the 'params' parameter of the peach-
function.

Local Control Code

The Local Control Code used to create the computational Project is shown below.

           1     def run_parameters(multip, jobs):
           2         import techila
           3         result = techila.peach(funcname = 'parameters_dist',
           4                                params = [multip, ''],
           5                                files = ['parameters_dist.py'],
           6                                peachvector = range(1, jobs + 1)
           7                                )
           8         print(result)
           9         return(result)

Apart from the parameters on lines 4 and 6 the parameters are similar as those explained in the earlier
example in Chapter 2.1. The parameters on lines 4 and 6 are explained below.

Line 4 contains the definition for the 'params' parameter. In this example, two input arguments will be
passed to the executable function. The first input argument ('multip') will be identical for all Jobs and
will correspond to the value passed as an input argument to the 'run_parameters' function. The
second input argument ('') will be replaced with a different 'peachvector' element for each
Job. These elements will be used to simulate the value of for-loop counter used in the locally executable
function.

Line 6 contains the definition for the 'peachvector'. In this example, the 'peachvector' will contain
elements from one (1) to the value of the 'jobs' parameter. As the '' notation was used in the
'params' parameter, these elements will be passed as input arguments to the executable function. Job
#1 will receive the first element (value 1), Job #2 will receive the second element (value 2) and so on.

The number of elements in the 'peachvector' will also define the number of Jobs in the Project. In this
example, the number elements will be the same as the value of the 'jobs' variable.

The value returned by the peach-function will be stored in the 'result' list. Each list element will
contain the result returned from one of the Jobs.

Worker Code

The function that will be executed on Workers (in file 'parameters_dist.py') is shown below.


                         def parameters_dist(multip, jobidx):
                             result = multip * jobidx
                             return(result)


The Local Control Code discussed earlier defined two parameters in the 'params' parameter, which will
be passed to the executable function ('parameters_dist') as input arguments. The first input
argument ('multip') will have the same value in all Jobs. The second input argument ('jobidx') will
be replaced with different 'peachvector' elements. The value of the 'jobidx' parameter will

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                17/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

fundamentally be used to simulate the value of the loop counter that was used in the locally executable
function.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 8.




Figure 8. Parameters listed in the 'params' parameter will be passed to executable function as input
arguments. The '' notation is used to transfer elements of the peachvector to the Worker
Code. The value of the 'jobs' variable is defined by the End-User and it is used to define the length
of the 'peachvector'. The value of the 'jobs' parameter therefore also defines the number of Jobs.

Creating the Project

To create the computational Project, change your current working directory (in Python) to the directory that
contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown
below:

execfile('run_parameters.py')
result = run_parameters(2,5)

If prompted, enter the password to your keystore file. The Project will be created and information on the
Project progress will be displayed in the console. The output generated by the commands should resemble
the one shown below in Figure 9.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                18/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013




Figure 9. Creating the computational Project. After all Jobs have been completed, the 'result' list
will be printed. Each element in the 'result' list will contain the value returned from one Job. The
Jobs will be extremely short as each Job consists of simply multiplying two integers.

The computational operations occurring during the Project are illustrated below in Figure 10.




Figure 10. Creating the project with the syntax shown in the figure will create a Project that
consists of five Jobs. The value of the first input argument is same for all Jobs. The second input
argument will be replaced with elements of the 'peachvector' and each Job will receive a different
element. The peach-function will return the Job results as a list called 'result' as defined in the
Local Control Code.

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                19/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


2.3       Transferring data files
This purpose of this example is to demonstrate:

          How to transfer data files to the Workers

Files can be transferred to Workers by using the 'datafiles' parameter of the peach-function. The
general syntax for using the parameter is explained below:

datafiles=['']

The  notation will need to be replaced with a comma separated list of files that you
wish to transfer to Workers. For example, the following syntax would transfer files called 'file1' and
'file2' (located in the current working directory) to the Workers.

datafiles=['file1','file2']

If the files that you wish to transfer to Workers are not located in your current working directory, the path of
the file needs to be defined. For example, the following syntax would transfer 'file1' from the current
working directory and 'file2' from the directory 'C:\temp' (backslashes '\' need to be escaped by
adding another backslash '\\').

datafiles=['file1','C:\\temp\\file2']

Please note that after the files have been transferred to Workers, all files will be stored in the same
temporary working directory with the executable code. In cases where you are using path definitions in
your file access methods, you might need to modify those commands to access the files from the current
working directory on the Worker.

Also note that the 'datafiles' parameter should only be used to transfer small files that change
frequently. This is because the files will be stored in the Parameter Bundle, which will be re-created
automatically when creating a new computational Project.

If you plan to transfer large files, or files that will not change frequently, it is advisable that you create a
separate Data Bundle to transfer the files. Instructions on how to use a Data Bundle to transfer data files
can be found in Chapter 3.6.

The material used in this example is located in the following folder in the Techila SDK:

          techila\examples\python\Tutorial\3_datafiles

Locally executable Python function

The locally executable Python script used in this example is shown in below.


      import csv

      def local_function():
          rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter=' '))
          contents=[]
          for row in rows:
              row_int = map(int,row)
              sum_row_int=sum(row_int)
              contents.append(sum_row_int)
          print('Sums of rows: ' , contents)
          return(contents)

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                20/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013



During the initial steps of the function, the file 'datafile.txt' will be opened and the rows will be read
and stored in the 'rows' variable. This file contains four rows of numbers, where each number is
separated by white space.

The function also contains a for-loop, which will be used to sum the values on each row and append the
summation result in the 'contents' list. After all rows have been processed the results will be displayed.

To execute the locally executable function, use the commands shown below:

execfile('local_function.py')
result=local_function()

The output generated by the function will be similar to the one shown below:




Figure 11. Executing the locally executable function. The result will be stored in the 'result' list,
which will contain four list elements. Each list element will contain the sum of values on one row.

Distributed version of the program

Each row in the 'datafile.txt' can be processed independently, meaning Jobs in the computational
Project can be configured to process different rows simultaneously. This can be achieved by transferring
the file 'datafile.txt' to Workers by using the 'datafiles' parameter of the peach-function.

Each Job can then be configured to process one of the rows in the data file. This can be achieved by using
the indexes of the 'peahcvector' (notation ('')) to index the file and process a different row
in each Job.

Local Control Code

The Local Control Code that is used to create the computational Project is shown below.

        1      def run_datafiles():
        2          import techila
        3          jobs = 4
        4          result = techila.peach(funcname = 'datafiles_dist',
        5                                 params = [''],
        6                                 files = ['datafiles_dist.py'],
        7                                 datafiles = ['datafile.txt'],
        8                                 peachvector = range(1, jobs + 1)
        9                                 )
        10         print('Sums of rows: ', result)
        11         return(result)

Apart from the parameters on lines 5 and 6 the parameters are similar as those explained in the earlier
example in Chapter 2.1. The parameters on lines 5 and 6 are explained below.


©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                21/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

Line 5 contains the definition for the 'params' parameter, which defines the input arguments that will be
passed to the executable function. In this example, the notation '' has been defined which
means index values of the 'peachvector' will be passed as an input arguments to the function. The
value of the notation will be zero (0) for Job #1, one (1) for Job #2 and so on. These index values will be
used to choose which row should be processed from the data file.

Line 6 contains the definition for the 'datafiles' parameter, which defines a list of files that will be
transferred to Workers. In this example, the file 'datafile.txt' has been defined and will be
transferred to Workers. After the file has been transferred to the Worker, it will be copied in the same
temporary working directory with the executable code.

Worker Code

The algorithm of the Worker Code used in this example is shown below.

 1     import csv
 2     def datafiles_dist(jobidx):
 3         rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter=' '))
 4         row = rows[jobidx]
 5         row_int = map(int,row)
 6         sum_row_int=sum(row_int)
 7         return(sum_row_int)

Line 1 imports the 'csv' package that will be used when reading the contents of the file. All packages
included in the standard Python distribution will be automatically available for use on the Workers.
Additional packages can be made available by using the 'bundleit' command as explained in Chapter
3.10.

Line 2 contains the start of the definition for the function that will be executed in each Job. The function
takes one input argument ('jobidx'), which will be replaced by the index values of the 'peachvector'
as explained earlier ('' notation in the Local Control Code). This input argument will be used
to select which row should be processed.

Lines 3 to 6 contain similar operations as performed inside the for-loop in the locally executable function
shown earlier. The operations will select the row that matches the value of the 'jobidx' variable and
sum the values on that row. This corresponds to the operations that are performed in one iteration of the
for-loop structure in the locally executable function.

Line 7 will return the summation result as the result from the computational Job.

The interaction between the Local Control Code and the Worker Code are shown below.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                22/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013




Figure 12. Parameters listed in the 'params' parameter will be given as input arguments to the
executable function. The 'datafile.txt' file will be transferred to the same temporary directory with
the executable code. The syntax for loading the 'datafile.txt' will be the same as in the locally
executable function.

Creating the computational project

To create the computational Project, change your current working directory (in Python) to the directory that
contains the example material for this example. After having navigated to the correct directory, create the
computational Project using commands shown below:

execfile('run_datafiles.py')
result = run_datafiles()

If prompted, enter the password to your keystore file. The Project will be created and information on the
Project progress will be displayed in the console. The number of Jobs in the Project will be automatically
fixed to four according to the value of the 'jobs' parameter defined in the Local Control Code. The
output generated by the commands should resemble the one shown below in Figure 13 .




Figure 13. Creating the computational Project. The 'result' list will contain four elements, one
element for each Job in the Project. As each Job consisted of summing the values on one row,
each list element will contain the same values as in the locally executable program.

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                23/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013


2.4       Multiple functions in an Python script
The purpose of this example is to demonstrate:

          How to define multiple functions in the Worker Code
          How to use different functions as the entry point when starting a Job on the Workers

The material used in this example is located in the following folder in the Techila SDK:

          techila\examples\python\Tutorial\4_multiplefunctions

Locally executable Python functions

The Python script containing the locally executable functions is shown below.


                                         def function1():
                                             return(1 + 1)

                                         def function2():
                                             return(10 * 10)


To execute the functions on your local computer, please evaluate the file using command:

execfile('local_multiple_functions.py')

After the functions have been defined, 'function1' can be executed using command:

result = function1()

When called, function1 will perform the summation 1+1 and return 2 as the result.

Respectively, the 'function2' can be executed with command:

result = function2()

When called, function2 will perform the multiplication 10*10 and return 100 as the result. This is
illustrated below in Figure 14.




Figure 14. Executing the local functions. Functions can be executed after the Python script
containing the function definitions has been evaluated with the 'execfile' function.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                24/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

Distributed version of the program

In the locally executable version of the example, the Python script contains two function definitions. These
functions were defined by using the 'execfile' function.

In the distributed version, the functions can be made available on Workers by including both function
definitions in the Python script that will be evaluated at the preliminary stage of the Job. This can be
achieved by using the 'files' parameter. As the file containing the functions will be evaluated before
any functions are called, this also means that either one of these functions can be used as an entry point
in the computational Job (values for the 'funcname' parameter).

The Local Control Code

The Local Control Code used to create the computational Project is shown below.


            def run_multi_function(funcname):
                import techila
                result = techila.peach(funcname = funcname,
                                       files = ['multi_function_dist.py'],
                                       peachvector = [1],
                                       )
                print(result)


The function 'run_multi_function' takes one input argument, which will be used to define the value
of the 'funcname' parameter. This means that the input argument will be used to determine, which
function will be executed during a Job.

The 'files' parameter lists one file called 'multifunction_dist.py', which contains the same
function definitions that were defined in the locally executable version of this example. This either one of
the functions ('function1' and 'function2') can be used as an entry point.

Worker Code

The file containing the Worker Code ('multi_function_dist.py') used in this example is shown
below.


                                         def function1():
                                             return(1 + 1)

                                         def function2():
                                             return(10 * 10)


As can be seen, the Worker Code contains the same function definitions as the locally executable version
of the program. The Python script containing the function definitions will be evaluated at the preliminary
stages of a computational Job, meaning both functions will be defined and can be called during a
computational Job.

This also means either function can also be used as the entry point when defining the 'funcname'
parameter in the Local Control Code.




©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
End-User Guide                      Techila with Python                                25/67

Author                                  Classification                      Date
Techila Technologies                    Public                              January 04, 2013

Creating the computational Project

To create the computational Project, change your current working directory (in Python) to the directory that
contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown
below:

execfile('run_multi_function.py')

After having evaluated the Local Control Code, a computational Project that executes 'function1' on
Workers can be created using command shown below:

result = run_multi_function('function1')

This will create a computational Project that consists of one (1) Job. The computational operations
occurring during the Project are illustrated below in Figure 15




Figure 15. The funcname parameter determines the name of the function that will be called in the
computational Job. In this example, 'function1' will be called.

Respectively, 'function2' can be executed on the Worker by using the command:

result = run_multi_function('function2')

The computational operations occurring during the Project are illustrated below in Figure 16.




Figure 16. A different entry point ('function2') can be defined by giving the applicable value to the
'funcname' parameter.



©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered
trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other
trademarks are the property of their respective owners.
Next part ... Cancel