TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

END-USER GUIDE TECHILA WITH PYTHON JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Disclaimer Techila Technologies Ltd. disclaims any and all warranties, express, implied or statutory regarding this document or the use of thereof by you to the full extent permitted by law.

Without limiting the generality of the foregoing, this document provided by Techila Technologies Ltd. in connection therewith are provided 'as-is' and without warranties of any kind, including, without limitation, any warranties of performance or implied warranties of merchantability, fitness for a particular purpose, title and noninfringement. Further, Techila Technologies Ltd. does not make, and has not made, any presentation or warranty that the document is accurate, complete, reliable, current, error-free, or free from harmful information. Limitation of Liability In no event shall Techila Technologies Ltd.

or any of its respective directors, officers, employees, or agents, be liable to you or any other person or entity, under any theory, including without limitation negligence, for damages of any kind arising from or related to the application of this document or any information, content, or materials in or accessible through this document, including, but not limited to, direct, indirect, actual, incidental, punitive, special or consequential damages, lost income, revenue or profits, lost or damaged data, or other commercial or economic loss, that result from your use of, or inability to use, this document, even if any of those persons or entities have been advised of the possibility of such damages or such damages are foreseeable.

Use of this document and copyright No part of this document may be used, reproduced, modified, or transmitted in any form or means without the prior written permission of Techila Technologies. This document and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. Copyright Techila Technologies 2013. All rights reserved.

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. Table of contents 1 Introduction . . 4 1.1 Installing the techila package . . 5 1.2 Example material . . 5 1.3 Naming convention of the python scripts . . 6 1.4 Python peach-function . . 6 1.5 Process flow in computational Projects . . 7 1.5.1 The peach-function . . 7 1.6 Techila environment variables . . 8 2 Peach tutorial examples . . 9 2.1 Executing a simple Python function on Workers . . 10 2.2 Using input parameters . . 14 2.3 Transferring data files . . 19 2.4 Multiple functions in an Python script . . 23 3 Peach feature examples .

. 26 3.1 Monte Carlo Pi with peach . . 28 3.2 Streaming & Callback Function . . 31 3.3 Job Input Files . . 35 3.4 Project Detaching . . 38 3.5 Iterative Projects . . 41 3.6 Data Bundles . . 44 3.7 Function handle . . 47 3.8 File Handler . . 49 3.9 Snapshots . . 52 3.10 Using custom Python packages in computational Projects . . 55 3.11 Precompiled binaries . . 61

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 4/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1 Introduction This document is intended for Techila End-Users who are using Python as their main development environment. If you are unfamiliar with Techila terminology or the operating principles of the Techila system, please see document "Techila Fundamentals" for more information.

The structure of this document is as follows: Chapter 1 contains important information regarding the installation of required Python packages that enable you to use Techila with Python. This Chapter also contains a brief introduction on the naming convention of the Python-scripts and introduces the peach-function, which is used for distributing computations from Python to the Techila environment.

Chapter 2 contains walkthroughs of simplistic example code samples that use the peach-function. The example material illustrates how to control the core features of the peach-function, including defining input arguments for the executable function, transferring data files to the Workers and calling different functions from the Python script that is evaluated on the Worker. After examining the material in this Chapter you should be able split a simple locally executable program into two pieces of code (Local Control Code and Worker Code), which in turn can be used to perform the computations in the Techila environment.

Chapter 3 contains walkthroughs of several examples that illustrate how to implement different features available in the peach-function. Each subchapter in this Chapter contains a walkthrough of an executable piece of code that illustrates how to implement one or more peach-function features. Each Chapter is named according to the feature that will be the focused on. After examining the material in this Chapter you should be able implement several features available in the peach-function in your own distributed application.

Please note that screenshots in this document are from a Windows 7 operating system.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 5/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1.1 Installing the techila package The 'techila' Python package is included in the Techila SDK and contains all Techila related Python functions required for creating computational Projects.

Please follow the steps below to install the package. 1. Launch a command prompt \ terminal 2. Change your current working directory to the following directory: cd \techila\lib\python 3. Install the 'techila' package using command: python setup.py install Figure 1.Installing the 'techila' package. 1.2 Example material The Python scripts containing the example material discussed in this document can be found in the Tutorial and Features folders in the Techila SDK. These folders contain subfolders, which contain the actual Python scripts that can be used to run the examples.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 6/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. techila examples python Tutorial Features Chapter 2 Chapter 3 Figure 2. The example material discussed in this document can be found in the in the 'python' folder in the Techila SDK 1.3 Naming convention of the python scripts The typical naming convention of Python scripts presented in this document is explained below:  Python scripts ending with '_dist' contain the Worker Code, which will be distributed to the Workers when the Local Control Code is executed.

 Python scripts beginning with 'run_' contain the Local Control Code, which will create the computational Project when executed locally on the End-User’s own computer.  Python scripts beginning with 'local_' contain locally executable code, which does not communicate with the Techila environment. Please note that some Python scripts and functions might be named differently, depending on their role in the computational Project. 1.4 Python peach-function The peach-function provides a simple interface that can be used to distribute Python programs or precompiled binaries. Each peach-function input argument is a named parameter, which refer to a computer language's support for function calls that state the name of each parameter within the function call itself.

A minimalistic Python peach-function syntax typically includes the following parameters:  funcname  params  files  peachvector  datafiles These parameters can be used to define input arguments for the executable function and transfer additional files to the Workers. An example of a peach-function syntax using these parameters is shown below:

End-User Guide Techila with Python 7/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. techila.peach(funcname = 'example_function', # Function executed on Workers params = [var1,var2], # Input arguments to the function files = ['codefile.py # Evaluated at the start of a Job datafiles = ['file1','file2 # Files transferred to Workers peachvector = [2,4,6,8,10] # Peachvector definition ) Examples and more detailed explanations of these parameters can be found in Chapter 2. General information on available peach-function parameters can also be displayed by executing the following commands in Python.

import techila help(techila.peach) 1.5 Process flow in computational Projects When a Project is created with the peach-function, each Job in a computational Project will have a separate Python session. When a Job is started on a Worker, functions and variables are loaded by evaluating the Python script that was defined in the 'files' parameter and by loading the parameters stored in the 'techila_peach_inputdata'file. These parameters will include the parameters defined in the 'params' peach-function parameter.

When a Job is started on a Worker, the 'peachclient.py' script (included in the 'techila' package) is called.

The 'peachclient.py' file is a Python script that acts as a wrapper for the Worker Code and is responsible for calling the executable function, passing input arguments to the function and returning the final computational results. This functionality is hidden from the End-User. The 'peachclient.py' will be used automatically in computational Projects created with peach-function. The 'peachclient.py' wrapper also sets a preliminary seed for the random number generator by using the Python seed() function from the package 'random'. Each Job in a computational Project will receive a unique random number seed based on the current system time and the 'jobidx' parameter, which will be different for each Job.

The preliminary random number seeding can be overridden by calling the seed() function in the Worker Code with the desired random seed.

1.5.1 The peach-function The list below contains some of the Python specific activities that are performed automatically when the peach-function is used to create a computational Project. 1. The peach-function is called locally on the End-Users computer 2. Python scripts listed in the 'files' parameter are transferred to Workers 3. Files listed in the 'datafiles' parameter are transferred to Workers 4. The 'peachclient.py' file is transferred to Workers 5. Input parameters listed in the 'params' parameter are stored in a file called 'techila_peach_inputdata', which is transferred to Workers.

6. The files listed in the 'files' and 'datafiles' parameters and the files 'techila_peach_inputdata'and 'peachclient.py' are copied to the temporary working directory on the Worker 7. The 'peachclient.py' wrapper is called on the Worker. 8. Variables stored in the file 'techila_peach_inputdata' are loaded 9. Files listed in the 'files' parameter are evaluated using the Python 'execfile' command 10. The '' notation is replaced with a 'peachvector' element

End-User Guide Techila with Python 8/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 11. The peachclient calls the function defined in the 'funcname' parameter with the input parameters defined in 'params'.

12. The peachclient saves the result in to a file, which is returned from the Worker to the End-User 13.

The peach-function reads the output file and stores the result in a list element (If a callback function is used, the result of the callback function is returned). 14. The entire list is returned by the peach-function. 1.6 Techila environment variables The table below contains descriptions of environment variables used by Techila. These environment variables can be used to define e.g. the path of the 'techila' directory on your computer or to define the version of Python runtime components used during computational Projects.

Environment variable Description Example value TECHILA_SDKROOT Defines the path of the root directory of the Techila SDK. C:\techila TECHILA_PYTHON_VERSION Defines the Python Runtime Bundle version that will be used in Projects. This parameter can be used e.g. if the Techila environment does not have a Python Runtime Bundle with the same version as your local Python environment. The version of the Runtime Bundle is defined using the syntax . (As returned by command 'sys.version_info' in Python) 273

End-User Guide Techila with Python 9/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2 Peach tutorial examples This Chapter contains four minimalistic examples on how to implement and control the core features of the peach-function. The example material discussed in this Chapter, including Python scripts and data files can be found in the subdirectories under the following folder in the Techila SDK:  techila\examples\python\Tutorial Each of the examples contains three pieces of code:  A script containing a locally executable Python function.

This function will be executed locally and will not communicate with the distributed computing environment in any way. This script is provided as reference material to illustrate what modifications are required to execute the computations in the Techila environment.

 A script containing the Local Control Code, which will be executed locally on the End-Users computer. This script contains the peach-function call, which will distribute the computations in the Worker Code to the distributed computing environment  A script containing the Worker Code, which will be executed on the Workers. This script contains the computationally intensive part of the locally executable script. Please note that the example material in this Chapter is only intended to illustrate the core mechanics related to distributing computation with the peach-function. More information on available features can be found in Chapter 3 and by executing the following commands in Python.

import techila help(techila.peach)

End-User Guide Techila with Python 10/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.1 Executing a simple Python function on Workers This example is intended to provide an introduction on distributed computing using Techila with Python using the peach-function.

The purpose of this example is to:  Demonstrate how to modify a simple, locally executable Python script that contains one function so that the computational operations will be performed in the Techila environment  Demonstrate the difference between Local Control Code and Worker Code in a Python environment  Demonstrate the basic syntax of the peach-function in a Python environment The material discussed in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\1_distribution Locally executable Python function The Python script 'local_function.py' contains one function called 'local_function', which consists of one for-loop.

The algorithm of the locally executable function used in this example is shown below.

def local_function(x): result = [] for j in range(x): result.append(1 + 1) return result The function takes one input argument, which defines the number of iterations that will be performed in the for-loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will be appended to the 'result' list. For example, when performing five iterations the 'result' list would contain the following values. To execute the locally executable function, use the commands shown below: execfile('local_function.py') result=local_function(5) The output generated by the locally executable program should resemble the one shown below in Figure 3.

Figure 3. Executing the locally executable function in Python. Number of iterations: 5 index 0 1 2 3 4 value 2 2 2 2 2

End-User Guide Techila with Python 11/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Distributed version of the program All arithmetic operations in the locally executable function are performed in the for-loop.

There are no recursive data dependencies between iterations, meaning that the all the iterations can be performed simultaneously. The iterations can be performed simultaneously by placing the arithmetic operations to a separate file (distribution_dist.py), which can be then transferred and executed on Workers. The file containing the Local Control Code ('run_distribution.py') will be used to create the computational Project. The Worker Code ('distribution_dist.py') will be transferred to the Workers where the file will automatically be evaluated (using 'execfile') at the preliminary stages of the Job.

After the file has been evaluated, the function 'distribution_dist' will be executed. The result returned by the function will be returned from the Job.

Local Control Code The Local Control Code used to create the computational Project is shown below. Line 1 contains the definition for the function 'run_distribution'. This function takes one input argument called 'jobcount' (integer). This variable will be used during the peach-function call to define the value of the 'jobs' parameter, which defines the number of Jobs in the Project. Line 2 imports the 'techila' package, which is required to use the Techila Python functions such as 'peach'.

Lines 3 to 6 contain the peach-function call, which will be used to create the computational Project.

After the computational Project has been completed, the results will be stored in the 'result' list. The number of list elements will be the same as the number of Jobs in the Project. Each list element will contain the result returned from one Job. The parameters of the peach-function call are explained below. funcname = 'distribution_dist' The 'funcname' parameter shown above defines that the function 'distributed_dist' will be executed in each Job. This function will be defined when the file 'distribution_dist.py' is evaluated as explained below.

files = 'distribution_dist.py' The parameter shown above defines that a file named 'distributed_dist_py' should be evaluated with the 'execfile' command at the preliminary stage of the computational Job. This evaluation will make all variables and functions defined in the file accessible during the computational Job. In this example, the file contains the definition for the 'distribution_dist' function. jobs = jobcount 1 def run_distribution(jobcount): 2 import techila 3 result = techila.peach(funcname = 'distribution_dist', 4 files = 'distribution_dist.py', 5 jobs=jobcount 6 ) 7 print(result) 8 return(result)

End-User Guide Techila with Python 12/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The 'jobs' parameter shown above defines the number of Jobs in the Project. In this example, the number of Jobs will be based on the value of the 'jobcount' variable.

The number of Jobs in the Project could also be defined by using the 'peachvector' parameter. For more information on how to define the 'peachvector', please see Chapter 2.2.

Line 7 contains a print command that will be used to print the 'result' list. Each element in the array will contain the result for one Job in the Project. Line 8 will return the 'result' list as the output. Worker Code The Worker Code that will be executed on the Workers is in the file called 'distribution_dist.py'. The content of the file is shown below. def distribution_dist(): result = 1 + 1 return(result) Operations performed in the Worker Code correspond to those performed during one iteration of the locally executable for-loop structure. Each Job will simply sum two integers (1+1) and store the value of the summation to the 'result' variable.

The 'result' variable will returned from the Job will be eventually received by the End-User as a one of the list elements returned by the peach-function in the Local Control Code.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 4. Figure 4. The names of Python scripts that will be evaluated using 'execfile' on the Worker are defined using the 'files' parameter. In this example, the file 'distribution_dist.py' will be transferred to all Workers and evaluated at the preliminary stages of the computational Job. The function that will be called is defined with the 'funcname' parameter. In this example, the function 'distribution_dist' will be called in each computational Job.

End-User Guide Techila with Python 13/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory containing the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_distribution.py') result = run_distribution(5) If prompted, enter the password to your keystore file. The computation Project will be created and information on the Project progress will be displayed in the console. The output generated by the commands should resemble the one shown below in Figure 5. Figure 5. Creating the computational Project in a Python session. The computational Jobs will be extremely short as each Job consists of simply summing up two integers; 1+1.

The computations occurring during the Project are illustrated below in Figure 6: Figure 6. The input argument passed to the function 'run_distribution' will be used to determine the number of Jobs in the Project. The same arithmetic operation, 1+1, is performed in each Job. Results are delivered back to the End-Users computer where they will be stored as list elements in the 'result' list.

End-User Guide Techila with Python 14/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.2 Using input parameters This purpose of this example is to demonstrate:  How to transfer input parameters to the executable function In this example, parameters will be transferred to the Workers and passed to the executable function as input arguments.

This will be performed by using the 'params' parameter of the peach-function. The general syntax for defining input arguments for the executable function is shown below: params = The '' notation will need to be replaced with a comma separated list of variables that you wish to pass as input arguments. For example, the following syntax would define two input arguments 'var1' and 'var2' for the function that will be executed on Workers. params = [var1, var2] The values of the input arguments will be read from your current Python session, meaning you can transfer any parameters that are currently defined.

Dynamic input arguments (arguments that will have a different value for each Job) can be passed to the executable function by using the '' notation. This notation will be replaced by a different element of the 'peachvector' in each Job. For example, the following syntax will pass three input arguments for the executable function. The first two input arguments ('var1' and 'var2') have the same values for all Jobs. The third input argument ('') will be replaced with 'peachvector' elements. Job #1 will receive the first element (value 2), Job #2 will receive the second element (value 4) and so on.

params = [var1, var2 ] , peachvector = [2,4,6,8,10] Please note that when defining the 'peachvector', the length of the 'peachvector' will also define the number of Jobs in the Project. With the example shown above, the Project would consist of five Jobs. Another parameter that can be used when performing parameter sweep type computations is the 'vecidx' parameter. When used, this notation will be replaced by the corresponding 'peachvector' index for each Job. For example, the following syntax will pass two input arguments to the executable function. The first input argument ) will be replaced with 'peachvector' elements and the second input argument ('') with the corresponding 'peachvector' index values.

params ] , peachvector = [2,4,6,8,10] The input arguments that would be passed to the executable functions during different Jobs are shown below.

Argument #1 ) Argument #2 ('') Job #1 2 0 Job #2 4 1 Job #3 6 2 Job #4 8 3 Job #5 10 4

End-User Guide Techila with Python 15/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The material discussed in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\2_parameters Locally executable Python function The algorithm for the locally executable function used in this example is shown below.

def local_function(multip, loops): result = [] for x in range(1, loops + 1): result.append(multip * x) print(result) return(result) This function takes two input arguments; 'multip' and 'loops'. The parameter 'loops' determines the number of iterations in the for-loop. The parameter 'multip' is a number, which will be multiplied with the iteration counter represented by 'x'. The result of this arithmetic operation will be appended to a list called 'result', which will be returned as the output value of the function. An example 'result' list (five iterations) is shown below. multip = 2; loops=5 index 0 1 2 3 4 result 2 4 6 8 10 The locally executable function can be executed using the commands shown below: execfile('local_function.py') result=local_function(2,5) After executing the commands, numerical values stored in the 'result' list will be displayed.

If you executed the function using the values shown above, the output generated by the function should resemble the one shown below in Figure 7 Figure 7. Executing the locally executable function in Python. Distributed version of the program All the computations in locally executable Python function are performed in the for-loop and there are no dependencies between the iterations. As a result of this, the locally executable program can be converted to a distributed version by extracting the arithmetic operation into a separate piece of code ('parameters_dist.py'), which will be executed on Workers.

End-User Guide Techila with Python 16/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The Local Control Code ('run_parameters.py') contains the peach-function call that will be used to create the computational Project. In order to perform similar operations in the Project as in the local version, input arguments need to be given to the executable function to simulate different iterations of the for-loop.

These input arguments will be transferred by the using the 'params' parameter of the peach- function.

Local Control Code The Local Control Code used to create the computational Project is shown below. Apart from the parameters on lines 4 and 6 the parameters are similar as those explained in the earlier example in Chapter 2.1. The parameters on lines 4 and 6 are explained below. Line 4 contains the definition for the 'params' parameter. In this example, two input arguments will be passed to the executable function. The first input argument ('multip') will be identical for all Jobs and will correspond to the value passed as an input argument to the 'run_parameters' function. The second input argument ) will be replaced with a different 'peachvector' element for each Job.

These elements will be used to simulate the value of for-loop counter used in the locally executable function.

Line 6 contains the definition for the 'peachvector'. In this example, the 'peachvector' will contain elements from one (1) to the value of the 'jobs' parameter. As the '' notation was used in the 'params' parameter, these elements will be passed as input arguments to the executable function. Job #1 will receive the first element (value 1), Job #2 will receive the second element (value 2) and so on. The number of elements in the 'peachvector' will also define the number of Jobs in the Project. In this example, the number elements will be the same as the value of the 'jobs' variable. The value returned by the peach-function will be stored in the 'result' list.

Each list element will contain the result returned from one of the Jobs.

Worker Code The function that will be executed on Workers (in file 'parameters_dist.py') is shown below. def parameters_dist(multip, jobidx): result = multip * jobidx return(result) The Local Control Code discussed earlier defined two parameters in the 'params' parameter, which will be passed to the executable function ('parameters_dist') as input arguments. The first input argument ('multip') will have the same value in all Jobs. The second input argument ('jobidx') will be replaced with different 'peachvector' elements. The value of the 'jobidx' parameter will 1 def run_parameters(multip, jobs): 2 import techila 3 result = techila.peach(funcname = 'parameters_dist', 4 params = [multip ] , 5 files = ['parameters_dist.py'], 6 peachvector = range(1, jobs + 1) 7 ) 8 print(result) 9 return(result)

End-User Guide Techila with Python 17/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. fundamentally be used to simulate the value of the loop counter that was used in the locally executable function.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 8.

Figure 8. Parameters listed in the 'params' parameter will be passed to executable function as input arguments. The '' notation is used to transfer elements of the peachvector to the Worker Code. The value of the 'jobs' variable is defined by the End-User and it is used to define the length of the 'peachvector'. The value of the 'jobs' parameter therefore also defines the number of Jobs. Creating the Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_parameters.py') result = run_parameters(2,5) If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. The output generated by the commands should resemble the one shown below in Figure 9.

End-User Guide Techila with Python 18/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 9. Creating the computational Project. After all Jobs have been completed, the 'result' list will be printed. Each element in the 'result' list will contain the value returned from one Job. The Jobs will be extremely short as each Job consists of simply multiplying two integers. The computational operations occurring during the Project are illustrated below in Figure 10.

Figure 10. Creating the project with the syntax shown in the figure will create a Project that consists of five Jobs. The value of the first input argument is same for all Jobs. The second input argument will be replaced with elements of the 'peachvector' and each Job will receive a different element. The peach-function will return the Job results as a list called 'result' as defined in the Local Control Code.

End-User Guide Techila with Python 19/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.3 Transferring data files This purpose of this example is to demonstrate:  How to transfer data files to the Workers Files can be transferred to Workers by using the 'datafiles' parameter of the peach-function.

The general syntax for using the parameter is explained below: datafiles=[''] The notation will need to be replaced with a comma separated list of files that you wish to transfer to Workers. For example, the following syntax would transfer files called 'file1' and 'file2' (located in the current working directory) to the Workers.

datafiles=['file1','file2'] If the files that you wish to transfer to Workers are not located in your current working directory, the path of the file needs to be defined. For example, the following syntax would transfer 'file1' from the current working directory and 'file2' from the directory 'C:\temp' (backslashes '\' need to be escaped by adding another backslash ) . datafiles=['file1','C:\\temp\\file2'] Please note that after the files have been transferred to Workers, all files will be stored in the same temporary working directory with the executable code. In cases where you are using path definitions in your file access methods, you might need to modify those commands to access the files from the current working directory on the Worker.

Also note that the 'datafiles' parameter should only be used to transfer small files that change frequently. This is because the files will be stored in the Parameter Bundle, which will be re-created automatically when creating a new computational Project. If you plan to transfer large files, or files that will not change frequently, it is advisable that you create a separate Data Bundle to transfer the files. Instructions on how to use a Data Bundle to transfer data files can be found in Chapter 3.6.

The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\3_datafiles Locally executable Python function The locally executable Python script used in this example is shown in below.

import csv def local_function(): rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter ) ) contents=[] for row in rows: row_int = map(int,row) sum_row_int=sum(row_int) contents.append(sum_row_int) print('Sums of rows , contents) return(contents)

End-User Guide Techila with Python 20/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. During the initial steps of the function, the file 'datafile.txt' will be opened and the rows will be read and stored in the 'rows' variable. This file contains four rows of numbers, where each number is separated by white space.

The function also contains a for-loop, which will be used to sum the values on each row and append the summation result in the 'contents' list. After all rows have been processed the results will be displayed. To execute the locally executable function, use the commands shown below: execfile('local_function.py') result=local_function() The output generated by the function will be similar to the one shown below: Figure 11. Executing the locally executable function. The result will be stored in the 'result' list, which will contain four list elements. Each list element will contain the sum of values on one row.

Distributed version of the program Each row in the 'datafile.txt' can be processed independently, meaning Jobs in the computational Project can be configured to process different rows simultaneously. This can be achieved by transferring the file 'datafile.txt' to Workers by using the 'datafiles' parameter of the peach-function. Each Job can then be configured to process one of the rows in the data file. This can be achieved by using the indexes of the 'peahcvector' (notation ) to index the file and process a different row in each Job.

Local Control Code The Local Control Code that is used to create the computational Project is shown below. 1 def run_datafiles(): 2 import techila 3 jobs = 4 4 result = techila.peach(funcname = 'datafiles_dist', 5 params ] , 6 files = ['datafiles_dist.py'], 7 datafiles = ['datafile.txt'], 8 peachvector = range(1, jobs + 1) 9 ) 10 print('Sums of rows: ', result) 11 return(result) Apart from the parameters on lines 5 and 6 the parameters are similar as those explained in the earlier example in Chapter 2.1. The parameters on lines 5 and 6 are explained below.

End-User Guide Techila with Python 21/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Line 5 contains the definition for the 'params' parameter, which defines the input arguments that will be passed to the executable function. In this example, the notation '' has been defined which means index values of the 'peachvector' will be passed as an input arguments to the function. The value of the notation will be zero (0) for Job #1, one (1) for Job #2 and so on.

These index values will be used to choose which row should be processed from the data file.

Line 6 contains the definition for the 'datafiles' parameter, which defines a list of files that will be transferred to Workers. In this example, the file 'datafile.txt' has been defined and will be transferred to Workers. After the file has been transferred to the Worker, it will be copied in the same temporary working directory with the executable code. Worker Code The algorithm of the Worker Code used in this example is shown below. 1 import csv 2 def datafiles_dist(jobidx): 3 rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter ) ) 4 row = rows[jobidx] 5 row_int = map(int,row) 6 sum_row_int=sum(row_int) 7 return(sum_row_int) Line 1 imports the 'csv' package that will be used when reading the contents of the file.

All packages included in the standard Python distribution will be automatically available for use on the Workers. Additional packages can be made available by using the 'bundleit' command as explained in Chapter 3.10.

Line 2 contains the start of the definition for the function that will be executed in each Job. The function takes one input argument ('jobidx'), which will be replaced by the index values of the 'peachvector' as explained earlier ('' notation in the Local Control Code). This input argument will be used to select which row should be processed. Lines 3 to 6 contain similar operations as performed inside the for-loop in the locally executable function shown earlier. The operations will select the row that matches the value of the 'jobidx' variable and sum the values on that row. This corresponds to the operations that are performed in one iteration of the for-loop structure in the locally executable function.

Line 7 will return the summation result as the result from the computational Job. The interaction between the Local Control Code and the Worker Code are shown below.

End-User Guide Techila with Python 22/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

Figure 12. Parameters listed in the 'params' parameter will be given as input arguments to the executable function. The 'datafile.txt' file will be transferred to the same temporary directory with the executable code. The syntax for loading the 'datafile.txt' will be the same as in the locally executable function.

Creating the computational project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_datafiles.py') result = run_datafiles() If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. The number of Jobs in the Project will be automatically fixed to four according to the value of the 'jobs' parameter defined in the Local Control Code.

The output generated by the commands should resemble the one shown below in Figure 13 . Figure 13. Creating the computational Project. The 'result' list will contain four elements, one element for each Job in the Project. As each Job consisted of summing the values on one row, each list element will contain the same values as in the locally executable program.

End-User Guide Techila with Python 23/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.4 Multiple functions in an Python script The purpose of this example is to demonstrate:  How to define multiple functions in the Worker Code  How to use different functions as the entry point when starting a Job on the Workers The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\4_multiplefunctions Locally executable Python functions The Python script containing the locally executable functions is shown below.

def function1(): return(1 + 1) def function2(): return(10 * 10) To execute the functions on your local computer, please evaluate the file using command: execfile('local_multiple_functions.py') After the functions have been defined, 'function1' can be executed using command: result = function1() When called, function1 will perform the summation 1+1 and return 2 as the result. Respectively, the 'function2' can be executed with command: result = function2() When called, function2 will perform the multiplication 10*10 and return 100 as the result. This is illustrated below in Figure 14.

Figure 14. Executing the local functions. Functions can be executed after the Python script containing the function definitions has been evaluated with the 'execfile' function.