TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

END-USER GUIDE TECHILA WITH PYTHON JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Disclaimer Techila Technologies Ltd. disclaims any and all warranties, express, implied or statutory regarding this document or the use of thereof by you to the full extent permitted by law.

Without limiting the generality of the foregoing, this document provided by Techila Technologies Ltd. in connection therewith are provided 'as-is' and without warranties of any kind, including, without limitation, any warranties of performance or implied warranties of merchantability, fitness for a particular purpose, title and noninfringement. Further, Techila Technologies Ltd. does not make, and has not made, any presentation or warranty that the document is accurate, complete, reliable, current, error-free, or free from harmful information. Limitation of Liability In no event shall Techila Technologies Ltd.

or any of its respective directors, officers, employees, or agents, be liable to you or any other person or entity, under any theory, including without limitation negligence, for damages of any kind arising from or related to the application of this document or any information, content, or materials in or accessible through this document, including, but not limited to, direct, indirect, actual, incidental, punitive, special or consequential damages, lost income, revenue or profits, lost or damaged data, or other commercial or economic loss, that result from your use of, or inability to use, this document, even if any of those persons or entities have been advised of the possibility of such damages or such damages are foreseeable.

Use of this document and copyright No part of this document may be used, reproduced, modified, or transmitted in any form or means without the prior written permission of Techila Technologies. This document and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. Copyright Techila Technologies 2013. All rights reserved.

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. Table of contents 1 Introduction . . 4 1.1 Installing the techila package . . 5 1.2 Example material . . 5 1.3 Naming convention of the python scripts . . 6 1.4 Python peach-function . . 6 1.5 Process flow in computational Projects . . 7 1.5.1 The peach-function . . 7 1.6 Techila environment variables . . 8 2 Peach tutorial examples . . 9 2.1 Executing a simple Python function on Workers . . 10 2.2 Using input parameters . . 14 2.3 Transferring data files . . 19 2.4 Multiple functions in an Python script . . 23 3 Peach feature examples .

. 26 3.1 Monte Carlo Pi with peach . . 28 3.2 Streaming & Callback Function . . 31 3.3 Job Input Files . . 35 3.4 Project Detaching . . 38 3.5 Iterative Projects . . 41 3.6 Data Bundles . . 44 3.7 Function handle . . 47 3.8 File Handler . . 49 3.9 Snapshots . . 52 3.10 Using custom Python packages in computational Projects . . 55 3.11 Precompiled binaries . . 61

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 4/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1 Introduction This document is intended for Techila End-Users who are using Python as their main development environment. If you are unfamiliar with Techila terminology or the operating principles of the Techila system, please see document "Techila Fundamentals" for more information.

The structure of this document is as follows: Chapter 1 contains important information regarding the installation of required Python packages that enable you to use Techila with Python. This Chapter also contains a brief introduction on the naming convention of the Python-scripts and introduces the peach-function, which is used for distributing computations from Python to the Techila environment.

Chapter 2 contains walkthroughs of simplistic example code samples that use the peach-function. The example material illustrates how to control the core features of the peach-function, including defining input arguments for the executable function, transferring data files to the Workers and calling different functions from the Python script that is evaluated on the Worker. After examining the material in this Chapter you should be able split a simple locally executable program into two pieces of code (Local Control Code and Worker Code), which in turn can be used to perform the computations in the Techila environment.

Chapter 3 contains walkthroughs of several examples that illustrate how to implement different features available in the peach-function. Each subchapter in this Chapter contains a walkthrough of an executable piece of code that illustrates how to implement one or more peach-function features. Each Chapter is named according to the feature that will be the focused on. After examining the material in this Chapter you should be able implement several features available in the peach-function in your own distributed application.

Please note that screenshots in this document are from a Windows 7 operating system.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 5/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1.1 Installing the techila package The 'techila' Python package is included in the Techila SDK and contains all Techila related Python functions required for creating computational Projects.

Please follow the steps below to install the package. 1. Launch a command prompt \ terminal 2. Change your current working directory to the following directory: cd \techila\lib\python 3. Install the 'techila' package using command: python setup.py install Figure 1.Installing the 'techila' package. 1.2 Example material The Python scripts containing the example material discussed in this document can be found in the Tutorial and Features folders in the Techila SDK. These folders contain subfolders, which contain the actual Python scripts that can be used to run the examples.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 6/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. techila examples python Tutorial Features Chapter 2 Chapter 3 Figure 2. The example material discussed in this document can be found in the in the 'python' folder in the Techila SDK 1.3 Naming convention of the python scripts The typical naming convention of Python scripts presented in this document is explained below:  Python scripts ending with '_dist' contain the Worker Code, which will be distributed to the Workers when the Local Control Code is executed.

 Python scripts beginning with 'run_' contain the Local Control Code, which will create the computational Project when executed locally on the End-User’s own computer.  Python scripts beginning with 'local_' contain locally executable code, which does not communicate with the Techila environment. Please note that some Python scripts and functions might be named differently, depending on their role in the computational Project. 1.4 Python peach-function The peach-function provides a simple interface that can be used to distribute Python programs or precompiled binaries. Each peach-function input argument is a named parameter, which refer to a computer language's support for function calls that state the name of each parameter within the function call itself.

A minimalistic Python peach-function syntax typically includes the following parameters:  funcname  params  files  peachvector  datafiles These parameters can be used to define input arguments for the executable function and transfer additional files to the Workers. An example of a peach-function syntax using these parameters is shown below:

End-User Guide Techila with Python 7/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. techila.peach(funcname = 'example_function', # Function executed on Workers params = [var1,var2], # Input arguments to the function files = ['codefile.py # Evaluated at the start of a Job datafiles = ['file1','file2 # Files transferred to Workers peachvector = [2,4,6,8,10] # Peachvector definition ) Examples and more detailed explanations of these parameters can be found in Chapter 2. General information on available peach-function parameters can also be displayed by executing the following commands in Python.

import techila help(techila.peach) 1.5 Process flow in computational Projects When a Project is created with the peach-function, each Job in a computational Project will have a separate Python session. When a Job is started on a Worker, functions and variables are loaded by evaluating the Python script that was defined in the 'files' parameter and by loading the parameters stored in the 'techila_peach_inputdata'file. These parameters will include the parameters defined in the 'params' peach-function parameter.

When a Job is started on a Worker, the 'peachclient.py' script (included in the 'techila' package) is called.

The 'peachclient.py' file is a Python script that acts as a wrapper for the Worker Code and is responsible for calling the executable function, passing input arguments to the function and returning the final computational results. This functionality is hidden from the End-User. The 'peachclient.py' will be used automatically in computational Projects created with peach-function. The 'peachclient.py' wrapper also sets a preliminary seed for the random number generator by using the Python seed() function from the package 'random'. Each Job in a computational Project will receive a unique random number seed based on the current system time and the 'jobidx' parameter, which will be different for each Job.

The preliminary random number seeding can be overridden by calling the seed() function in the Worker Code with the desired random seed.

1.5.1 The peach-function The list below contains some of the Python specific activities that are performed automatically when the peach-function is used to create a computational Project. 1. The peach-function is called locally on the End-Users computer 2. Python scripts listed in the 'files' parameter are transferred to Workers 3. Files listed in the 'datafiles' parameter are transferred to Workers 4. The 'peachclient.py' file is transferred to Workers 5. Input parameters listed in the 'params' parameter are stored in a file called 'techila_peach_inputdata', which is transferred to Workers.

6. The files listed in the 'files' and 'datafiles' parameters and the files 'techila_peach_inputdata'and 'peachclient.py' are copied to the temporary working directory on the Worker 7. The 'peachclient.py' wrapper is called on the Worker. 8. Variables stored in the file 'techila_peach_inputdata' are loaded 9. Files listed in the 'files' parameter are evaluated using the Python 'execfile' command 10. The '' notation is replaced with a 'peachvector' element

End-User Guide Techila with Python 8/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 11. The peachclient calls the function defined in the 'funcname' parameter with the input parameters defined in 'params'.

12. The peachclient saves the result in to a file, which is returned from the Worker to the End-User 13.

The peach-function reads the output file and stores the result in a list element (If a callback function is used, the result of the callback function is returned). 14. The entire list is returned by the peach-function. 1.6 Techila environment variables The table below contains descriptions of environment variables used by Techila. These environment variables can be used to define e.g. the path of the 'techila' directory on your computer or to define the version of Python runtime components used during computational Projects.

Environment variable Description Example value TECHILA_SDKROOT Defines the path of the root directory of the Techila SDK. C:\techila TECHILA_PYTHON_VERSION Defines the Python Runtime Bundle version that will be used in Projects. This parameter can be used e.g. if the Techila environment does not have a Python Runtime Bundle with the same version as your local Python environment. The version of the Runtime Bundle is defined using the syntax . (As returned by command 'sys.version_info' in Python) 273

End-User Guide Techila with Python 9/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2 Peach tutorial examples This Chapter contains four minimalistic examples on how to implement and control the core features of the peach-function. The example material discussed in this Chapter, including Python scripts and data files can be found in the subdirectories under the following folder in the Techila SDK:  techila\examples\python\Tutorial Each of the examples contains three pieces of code:  A script containing a locally executable Python function.

This function will be executed locally and will not communicate with the distributed computing environment in any way. This script is provided as reference material to illustrate what modifications are required to execute the computations in the Techila environment.

 A script containing the Local Control Code, which will be executed locally on the End-Users computer. This script contains the peach-function call, which will distribute the computations in the Worker Code to the distributed computing environment  A script containing the Worker Code, which will be executed on the Workers. This script contains the computationally intensive part of the locally executable script. Please note that the example material in this Chapter is only intended to illustrate the core mechanics related to distributing computation with the peach-function. More information on available features can be found in Chapter 3 and by executing the following commands in Python.

import techila help(techila.peach)

End-User Guide Techila with Python 10/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.1 Executing a simple Python function on Workers This example is intended to provide an introduction on distributed computing using Techila with Python using the peach-function.

The purpose of this example is to:  Demonstrate how to modify a simple, locally executable Python script that contains one function so that the computational operations will be performed in the Techila environment  Demonstrate the difference between Local Control Code and Worker Code in a Python environment  Demonstrate the basic syntax of the peach-function in a Python environment The material discussed in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\1_distribution Locally executable Python function The Python script 'local_function.py' contains one function called 'local_function', which consists of one for-loop.

The algorithm of the locally executable function used in this example is shown below.

def local_function(x): result = [] for j in range(x): result.append(1 + 1) return result The function takes one input argument, which defines the number of iterations that will be performed in the for-loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will be appended to the 'result' list. For example, when performing five iterations the 'result' list would contain the following values. To execute the locally executable function, use the commands shown below: execfile('local_function.py') result=local_function(5) The output generated by the locally executable program should resemble the one shown below in Figure 3.

Figure 3. Executing the locally executable function in Python. Number of iterations: 5 index 0 1 2 3 4 value 2 2 2 2 2

End-User Guide Techila with Python 11/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Distributed version of the program All arithmetic operations in the locally executable function are performed in the for-loop.

There are no recursive data dependencies between iterations, meaning that the all the iterations can be performed simultaneously. The iterations can be performed simultaneously by placing the arithmetic operations to a separate file (distribution_dist.py), which can be then transferred and executed on Workers. The file containing the Local Control Code ('run_distribution.py') will be used to create the computational Project. The Worker Code ('distribution_dist.py') will be transferred to the Workers where the file will automatically be evaluated (using 'execfile') at the preliminary stages of the Job.

After the file has been evaluated, the function 'distribution_dist' will be executed. The result returned by the function will be returned from the Job.

Local Control Code The Local Control Code used to create the computational Project is shown below. Line 1 contains the definition for the function 'run_distribution'. This function takes one input argument called 'jobcount' (integer). This variable will be used during the peach-function call to define the value of the 'jobs' parameter, which defines the number of Jobs in the Project. Line 2 imports the 'techila' package, which is required to use the Techila Python functions such as 'peach'.

Lines 3 to 6 contain the peach-function call, which will be used to create the computational Project.

After the computational Project has been completed, the results will be stored in the 'result' list. The number of list elements will be the same as the number of Jobs in the Project. Each list element will contain the result returned from one Job. The parameters of the peach-function call are explained below. funcname = 'distribution_dist' The 'funcname' parameter shown above defines that the function 'distributed_dist' will be executed in each Job. This function will be defined when the file 'distribution_dist.py' is evaluated as explained below.

files = 'distribution_dist.py' The parameter shown above defines that a file named 'distributed_dist_py' should be evaluated with the 'execfile' command at the preliminary stage of the computational Job. This evaluation will make all variables and functions defined in the file accessible during the computational Job. In this example, the file contains the definition for the 'distribution_dist' function. jobs = jobcount 1 def run_distribution(jobcount): 2 import techila 3 result = techila.peach(funcname = 'distribution_dist', 4 files = 'distribution_dist.py', 5 jobs=jobcount 6 ) 7 print(result) 8 return(result)

End-User Guide Techila with Python 12/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The 'jobs' parameter shown above defines the number of Jobs in the Project. In this example, the number of Jobs will be based on the value of the 'jobcount' variable.

The number of Jobs in the Project could also be defined by using the 'peachvector' parameter. For more information on how to define the 'peachvector', please see Chapter 2.2.

Line 7 contains a print command that will be used to print the 'result' list. Each element in the array will contain the result for one Job in the Project. Line 8 will return the 'result' list as the output. Worker Code The Worker Code that will be executed on the Workers is in the file called 'distribution_dist.py'. The content of the file is shown below. def distribution_dist(): result = 1 + 1 return(result) Operations performed in the Worker Code correspond to those performed during one iteration of the locally executable for-loop structure. Each Job will simply sum two integers (1+1) and store the value of the summation to the 'result' variable.

The 'result' variable will returned from the Job will be eventually received by the End-User as a one of the list elements returned by the peach-function in the Local Control Code.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 4. Figure 4. The names of Python scripts that will be evaluated using 'execfile' on the Worker are defined using the 'files' parameter. In this example, the file 'distribution_dist.py' will be transferred to all Workers and evaluated at the preliminary stages of the computational Job. The function that will be called is defined with the 'funcname' parameter. In this example, the function 'distribution_dist' will be called in each computational Job.

End-User Guide Techila with Python 13/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory containing the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_distribution.py') result = run_distribution(5) If prompted, enter the password to your keystore file. The computation Project will be created and information on the Project progress will be displayed in the console. The output generated by the commands should resemble the one shown below in Figure 5. Figure 5. Creating the computational Project in a Python session. The computational Jobs will be extremely short as each Job consists of simply summing up two integers; 1+1.

The computations occurring during the Project are illustrated below in Figure 6: Figure 6. The input argument passed to the function 'run_distribution' will be used to determine the number of Jobs in the Project. The same arithmetic operation, 1+1, is performed in each Job. Results are delivered back to the End-Users computer where they will be stored as list elements in the 'result' list.

End-User Guide Techila with Python 14/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.2 Using input parameters This purpose of this example is to demonstrate:  How to transfer input parameters to the executable function In this example, parameters will be transferred to the Workers and passed to the executable function as input arguments.

This will be performed by using the 'params' parameter of the peach-function. The general syntax for defining input arguments for the executable function is shown below: params = The '' notation will need to be replaced with a comma separated list of variables that you wish to pass as input arguments. For example, the following syntax would define two input arguments 'var1' and 'var2' for the function that will be executed on Workers. params = [var1, var2] The values of the input arguments will be read from your current Python session, meaning you can transfer any parameters that are currently defined.

Dynamic input arguments (arguments that will have a different value for each Job) can be passed to the executable function by using the '' notation. This notation will be replaced by a different element of the 'peachvector' in each Job. For example, the following syntax will pass three input arguments for the executable function. The first two input arguments ('var1' and 'var2') have the same values for all Jobs. The third input argument ('') will be replaced with 'peachvector' elements. Job #1 will receive the first element (value 2), Job #2 will receive the second element (value 4) and so on.

params = [var1, var2 ] , peachvector = [2,4,6,8,10] Please note that when defining the 'peachvector', the length of the 'peachvector' will also define the number of Jobs in the Project. With the example shown above, the Project would consist of five Jobs. Another parameter that can be used when performing parameter sweep type computations is the 'vecidx' parameter. When used, this notation will be replaced by the corresponding 'peachvector' index for each Job. For example, the following syntax will pass two input arguments to the executable function. The first input argument ) will be replaced with 'peachvector' elements and the second input argument ('') with the corresponding 'peachvector' index values.

params ] , peachvector = [2,4,6,8,10] The input arguments that would be passed to the executable functions during different Jobs are shown below.

Argument #1 ) Argument #2 ('') Job #1 2 0 Job #2 4 1 Job #3 6 2 Job #4 8 3 Job #5 10 4

End-User Guide Techila with Python 15/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The material discussed in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\2_parameters Locally executable Python function The algorithm for the locally executable function used in this example is shown below.

def local_function(multip, loops): result = [] for x in range(1, loops + 1): result.append(multip * x) print(result) return(result) This function takes two input arguments; 'multip' and 'loops'. The parameter 'loops' determines the number of iterations in the for-loop. The parameter 'multip' is a number, which will be multiplied with the iteration counter represented by 'x'. The result of this arithmetic operation will be appended to a list called 'result', which will be returned as the output value of the function. An example 'result' list (five iterations) is shown below. multip = 2; loops=5 index 0 1 2 3 4 result 2 4 6 8 10 The locally executable function can be executed using the commands shown below: execfile('local_function.py') result=local_function(2,5) After executing the commands, numerical values stored in the 'result' list will be displayed.

If you executed the function using the values shown above, the output generated by the function should resemble the one shown below in Figure 7 Figure 7. Executing the locally executable function in Python. Distributed version of the program All the computations in locally executable Python function are performed in the for-loop and there are no dependencies between the iterations. As a result of this, the locally executable program can be converted to a distributed version by extracting the arithmetic operation into a separate piece of code ('parameters_dist.py'), which will be executed on Workers.

End-User Guide Techila with Python 16/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The Local Control Code ('run_parameters.py') contains the peach-function call that will be used to create the computational Project. In order to perform similar operations in the Project as in the local version, input arguments need to be given to the executable function to simulate different iterations of the for-loop.

These input arguments will be transferred by the using the 'params' parameter of the peach- function.

Local Control Code The Local Control Code used to create the computational Project is shown below. Apart from the parameters on lines 4 and 6 the parameters are similar as those explained in the earlier example in Chapter 2.1. The parameters on lines 4 and 6 are explained below. Line 4 contains the definition for the 'params' parameter. In this example, two input arguments will be passed to the executable function. The first input argument ('multip') will be identical for all Jobs and will correspond to the value passed as an input argument to the 'run_parameters' function. The second input argument ) will be replaced with a different 'peachvector' element for each Job.

These elements will be used to simulate the value of for-loop counter used in the locally executable function.

Line 6 contains the definition for the 'peachvector'. In this example, the 'peachvector' will contain elements from one (1) to the value of the 'jobs' parameter. As the '' notation was used in the 'params' parameter, these elements will be passed as input arguments to the executable function. Job #1 will receive the first element (value 1), Job #2 will receive the second element (value 2) and so on. The number of elements in the 'peachvector' will also define the number of Jobs in the Project. In this example, the number elements will be the same as the value of the 'jobs' variable. The value returned by the peach-function will be stored in the 'result' list.

Each list element will contain the result returned from one of the Jobs.

Worker Code The function that will be executed on Workers (in file 'parameters_dist.py') is shown below. def parameters_dist(multip, jobidx): result = multip * jobidx return(result) The Local Control Code discussed earlier defined two parameters in the 'params' parameter, which will be passed to the executable function ('parameters_dist') as input arguments. The first input argument ('multip') will have the same value in all Jobs. The second input argument ('jobidx') will be replaced with different 'peachvector' elements. The value of the 'jobidx' parameter will 1 def run_parameters(multip, jobs): 2 import techila 3 result = techila.peach(funcname = 'parameters_dist', 4 params = [multip ] , 5 files = ['parameters_dist.py'], 6 peachvector = range(1, jobs + 1) 7 ) 8 print(result) 9 return(result)

End-User Guide Techila with Python 17/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. fundamentally be used to simulate the value of the loop counter that was used in the locally executable function.

The interaction between the Local Control Code and the Worker Code is illustrated below in Figure 8.

Figure 8. Parameters listed in the 'params' parameter will be passed to executable function as input arguments. The '' notation is used to transfer elements of the peachvector to the Worker Code. The value of the 'jobs' variable is defined by the End-User and it is used to define the length of the 'peachvector'. The value of the 'jobs' parameter therefore also defines the number of Jobs. Creating the Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_parameters.py') result = run_parameters(2,5) If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. The output generated by the commands should resemble the one shown below in Figure 9.

End-User Guide Techila with Python 18/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 9. Creating the computational Project. After all Jobs have been completed, the 'result' list will be printed. Each element in the 'result' list will contain the value returned from one Job. The Jobs will be extremely short as each Job consists of simply multiplying two integers. The computational operations occurring during the Project are illustrated below in Figure 10.

Figure 10. Creating the project with the syntax shown in the figure will create a Project that consists of five Jobs. The value of the first input argument is same for all Jobs. The second input argument will be replaced with elements of the 'peachvector' and each Job will receive a different element. The peach-function will return the Job results as a list called 'result' as defined in the Local Control Code.

End-User Guide Techila with Python 19/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.3 Transferring data files This purpose of this example is to demonstrate:  How to transfer data files to the Workers Files can be transferred to Workers by using the 'datafiles' parameter of the peach-function.

The general syntax for using the parameter is explained below: datafiles=[''] The notation will need to be replaced with a comma separated list of files that you wish to transfer to Workers. For example, the following syntax would transfer files called 'file1' and 'file2' (located in the current working directory) to the Workers.

datafiles=['file1','file2'] If the files that you wish to transfer to Workers are not located in your current working directory, the path of the file needs to be defined. For example, the following syntax would transfer 'file1' from the current working directory and 'file2' from the directory 'C:\temp' (backslashes '\' need to be escaped by adding another backslash ) . datafiles=['file1','C:\\temp\\file2'] Please note that after the files have been transferred to Workers, all files will be stored in the same temporary working directory with the executable code. In cases where you are using path definitions in your file access methods, you might need to modify those commands to access the files from the current working directory on the Worker.

Also note that the 'datafiles' parameter should only be used to transfer small files that change frequently. This is because the files will be stored in the Parameter Bundle, which will be re-created automatically when creating a new computational Project. If you plan to transfer large files, or files that will not change frequently, it is advisable that you create a separate Data Bundle to transfer the files. Instructions on how to use a Data Bundle to transfer data files can be found in Chapter 3.6.

The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\3_datafiles Locally executable Python function The locally executable Python script used in this example is shown in below.

import csv def local_function(): rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter ) ) contents=[] for row in rows: row_int = map(int,row) sum_row_int=sum(row_int) contents.append(sum_row_int) print('Sums of rows , contents) return(contents)

End-User Guide Techila with Python 20/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. During the initial steps of the function, the file 'datafile.txt' will be opened and the rows will be read and stored in the 'rows' variable. This file contains four rows of numbers, where each number is separated by white space.

The function also contains a for-loop, which will be used to sum the values on each row and append the summation result in the 'contents' list. After all rows have been processed the results will be displayed. To execute the locally executable function, use the commands shown below: execfile('local_function.py') result=local_function() The output generated by the function will be similar to the one shown below: Figure 11. Executing the locally executable function. The result will be stored in the 'result' list, which will contain four list elements. Each list element will contain the sum of values on one row.

Distributed version of the program Each row in the 'datafile.txt' can be processed independently, meaning Jobs in the computational Project can be configured to process different rows simultaneously. This can be achieved by transferring the file 'datafile.txt' to Workers by using the 'datafiles' parameter of the peach-function. Each Job can then be configured to process one of the rows in the data file. This can be achieved by using the indexes of the 'peahcvector' (notation ) to index the file and process a different row in each Job.

Local Control Code The Local Control Code that is used to create the computational Project is shown below. 1 def run_datafiles(): 2 import techila 3 jobs = 4 4 result = techila.peach(funcname = 'datafiles_dist', 5 params ] , 6 files = ['datafiles_dist.py'], 7 datafiles = ['datafile.txt'], 8 peachvector = range(1, jobs + 1) 9 ) 10 print('Sums of rows: ', result) 11 return(result) Apart from the parameters on lines 5 and 6 the parameters are similar as those explained in the earlier example in Chapter 2.1. The parameters on lines 5 and 6 are explained below.

End-User Guide Techila with Python 21/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Line 5 contains the definition for the 'params' parameter, which defines the input arguments that will be passed to the executable function. In this example, the notation '' has been defined which means index values of the 'peachvector' will be passed as an input arguments to the function. The value of the notation will be zero (0) for Job #1, one (1) for Job #2 and so on.

These index values will be used to choose which row should be processed from the data file.

Line 6 contains the definition for the 'datafiles' parameter, which defines a list of files that will be transferred to Workers. In this example, the file 'datafile.txt' has been defined and will be transferred to Workers. After the file has been transferred to the Worker, it will be copied in the same temporary working directory with the executable code. Worker Code The algorithm of the Worker Code used in this example is shown below. 1 import csv 2 def datafiles_dist(jobidx): 3 rows = list(csv.reader(open('datafile.txt', 'rb'), delimiter ) ) 4 row = rows[jobidx] 5 row_int = map(int,row) 6 sum_row_int=sum(row_int) 7 return(sum_row_int) Line 1 imports the 'csv' package that will be used when reading the contents of the file.

All packages included in the standard Python distribution will be automatically available for use on the Workers. Additional packages can be made available by using the 'bundleit' command as explained in Chapter 3.10.

Line 2 contains the start of the definition for the function that will be executed in each Job. The function takes one input argument ('jobidx'), which will be replaced by the index values of the 'peachvector' as explained earlier ('' notation in the Local Control Code). This input argument will be used to select which row should be processed. Lines 3 to 6 contain similar operations as performed inside the for-loop in the locally executable function shown earlier. The operations will select the row that matches the value of the 'jobidx' variable and sum the values on that row. This corresponds to the operations that are performed in one iteration of the for-loop structure in the locally executable function.

Line 7 will return the summation result as the result from the computational Job. The interaction between the Local Control Code and the Worker Code are shown below.

End-User Guide Techila with Python 22/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

Figure 12. Parameters listed in the 'params' parameter will be given as input arguments to the executable function. The 'datafile.txt' file will be transferred to the same temporary directory with the executable code. The syntax for loading the 'datafile.txt' will be the same as in the locally executable function.

Creating the computational project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_datafiles.py') result = run_datafiles() If prompted, enter the password to your keystore file. The Project will be created and information on the Project progress will be displayed in the console. The number of Jobs in the Project will be automatically fixed to four according to the value of the 'jobs' parameter defined in the Local Control Code.

The output generated by the commands should resemble the one shown below in Figure 13 . Figure 13. Creating the computational Project. The 'result' list will contain four elements, one element for each Job in the Project. As each Job consisted of summing the values on one row, each list element will contain the same values as in the locally executable program.

End-User Guide Techila with Python 23/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.4 Multiple functions in an Python script The purpose of this example is to demonstrate:  How to define multiple functions in the Worker Code  How to use different functions as the entry point when starting a Job on the Workers The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\4_multiplefunctions Locally executable Python functions The Python script containing the locally executable functions is shown below.

def function1(): return(1 + 1) def function2(): return(10 * 10) To execute the functions on your local computer, please evaluate the file using command: execfile('local_multiple_functions.py') After the functions have been defined, 'function1' can be executed using command: result = function1() When called, function1 will perform the summation 1+1 and return 2 as the result. Respectively, the 'function2' can be executed with command: result = function2() When called, function2 will perform the multiplication 10*10 and return 100 as the result. This is illustrated below in Figure 14.

Figure 14. Executing the local functions. Functions can be executed after the Python script containing the function definitions has been evaluated with the 'execfile' function.

End-User Guide Techila with Python 24/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

Distributed version of the program In the locally executable version of the example, the Python script contains two function definitions. These functions were defined by using the 'execfile' function.

In the distributed version, the functions can be made available on Workers by including both function definitions in the Python script that will be evaluated at the preliminary stage of the Job. This can be achieved by using the 'files' parameter. As the file containing the functions will be evaluated before any functions are called, this also means that either one of these functions can be used as an entry point in the computational Job (values for the 'funcname' parameter). The Local Control Code The Local Control Code used to create the computational Project is shown below. def run_multi_function(funcname): import techila result = techila.peach(funcname = funcname, files = ['multi_function_dist.py'], peachvector = [1], ) print(result) The function 'run_multi_function' takes one input argument, which will be used to define the value of the 'funcname' parameter.

This means that the input argument will be used to determine, which function will be executed during a Job.

The 'files' parameter lists one file called 'multifunction_dist.py', which contains the same function definitions that were defined in the locally executable version of this example. This either one of the functions ('function1' and 'function2') can be used as an entry point. Worker Code The file containing the Worker Code ('multi_function_dist.py') used in this example is shown below. def function1(): return(1 + 1) def function2(): return(10 * 10) As can be seen, the Worker Code contains the same function definitions as the locally executable version of the program. The Python script containing the function definitions will be evaluated at the preliminary stages of a computational Job, meaning both functions will be defined and can be called during a computational Job.

This also means either function can also be used as the entry point when defining the 'funcname' parameter in the Local Control Code.

End-User Guide Techila with Python 25/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_multi_function.py') After having evaluated the Local Control Code, a computational Project that executes 'function1' on Workers can be created using command shown below: result = run_multi_function('function1') This will create a computational Project that consists of one (1) Job. The computational operations occurring during the Project are illustrated below in Figure 15 Figure 15. The funcname parameter determines the name of the function that will be called in the computational Job.

In this example, 'function1' will be called. Respectively, 'function2' can be executed on the Worker by using the command: result = run_multi_function('function2') The computational operations occurring during the Project are illustrated below in Figure 16. Figure 16. A different entry point ('function2') can be defined by giving the applicable value to the 'funcname' parameter.

End-User Guide Techila with Python 26/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3 Peach feature examples The basic methodology and syntax of distributing computations with the peach-function was illustrated in the Tutorial in Chapter 2. In addition to the basic mechanics, the peach-function offers a wide range of optional features.

These features are illustrated in the examples discussed in Chapters 3.1 to 3.11. Several of the examples will use the approximation of Pi with a Monte Carlo method as a framework when demonstrating the mechanics of different features. The basic implementation for the Monte Carlo method can be found in Chapter 3.1. This implementation and can be used as a reference point to see what changes are required for implementing different features.

The example material discussed this Chapter, including Python scripts and data files can be found in the example specific directories under the following directory in the Techila SDK:  techila\examples\python\Features\ Please note that the example material discussed in this Chapter does not contain examples on all available peach-function features. For a complete list on available features, execute the following command in Python: import techila help(techila.peach) Approximating the value of Pi with a Monte Carlo Method – Basic Principle The Monte Carlo method is based on statistical simulation where random numbers are used to model and solve a computational problem.

This method can also be used to approximate the value of Pi with the help of a unit circle and a random number generator.

The area of the unit circle shown in Figure 17 is determined by the equation and the area of the square surrounding it by the equation . This means the ratio of areas is defined as follows: When a random point will be generated, it can be located within or outside the unit circle. When a large number of points are being generated with a reliable random number generator, they will be spread evenly over the square. As more and more points are generated, the ratio of points within circle compared to the total number of points starts to approximate the ratio of the two areas.

Figure 17.

A unit circle has an area equal to . The square surrounding a unit circle has an area equal to 4.

End-User Guide Techila with Python 27/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. For example, in a simulation of 1000 random points, the typical number of points within the circle is approximately 785. This means that the value of Pi is calculated in the following way.

Algorithmic approaches are usually done only using ¼ of a circle with a radius of 1. This is simply because of the fact that number generating algorithms on many platforms generate random numbers with a uniform(0,1) distribution. This does not change the approximation procedure, because the ratios of the areas remain the same.

End-User Guide Techila with Python 28/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.1 Monte Carlo Pi with peach The purpose of this example is to demonstrate:  How to approximate the value of Pi using Monte Carlo method  Converting a locally implemented Monte Carlo method to a distributed version The material used in this example is located in the following folder in the Techila SDK:  techila\examples\Python\Features\basic_monte_carlo_pi Locally executable function The locally executable function for approximating the value of Pi used in this example is shown below.

import random def local_function(loops): count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 pivalue = 4 * float(count) / loops print 'The approximated value of Pi is:', pivalue return(pivalue) The function takes one input argument called 'loops', which determines the number of iterations in the for-loop. During each iteration, two random numbers will be generated, which will be used as the coordinates of the random point. The coordinates of the point are then used to calculate the distance of the point from the centre of the unit circle.

If the distance is less than one, the point is located within the unit circle and the counter is incremented by one. After all iterations have been completed, the value of Pi will be calculated.

To execute the function locally on your computer, use the commands shown below: execfile('local_function.py') result = local_function(10000000) This will approximate the value of Pi using 10,000,000 randomly generated points. The operation will take approximately one minute, depending on your CPU. If you wish to perform a shorter approximation, reduce the number of random points generated to e.g. 1,000,000. After the approximation is completed, the approximated value of Pi will be displayed as shown below: Figure 18. Executing the local version of the Monte Carlo Pi example. The time required to execute the program will depend on the number of iterations performed.

End-User Guide Techila with Python 29/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Distributed version of program The computationally intensive part in the Monte Carlo method is the random number sampling, which is performed inside the for-loop in the locally executable function.

There are no dependencies between the iterations, meaning that the sampling process can be divided into a separate function and executed simultaneously on Workers.

Note that the seed of the random number generator is initialized automatically on the Workers by the peachclient as explained in Chapter 1.4. If you wish to use a different seeding method, please seed the random number generator directly in the Worker Code. Local Control Code The Local Control Code used in this example to create the computational Project is shown below. import techila def run_mcpi(jobcount, loops): result = techila.peach(funcname = 'mcpi_dist', params = [loops], files = ['mcpi_dist.py'], jobs=jobcount, ) result = 4 * float(sum(result ( jobcount * loops) print 'The approximated value of Pi is:', result return(result) The file contains one function called 'run_mcpi', which takes two input arguments: 'jobcount' and 'loops'.

The 'jobcount' parameter will be used to define the value for the 'jobs' parameter (in the peach-function call) and will define the number of Jobs in the Project. The 'loops' parameter will be passed as an input argument (by using 'params') to the 'mcpi_dist' function that will be executed on Workers.

After all Jobs have been completed, the peach-function will return a list called 'result'. Each list element will correspond to the result generated in one Job and will contain the number of points within the unitary circle. These values will be combined and used to calculate the approximate value of Pi. Worker Code The Worker Code used in this example is shown below. import random def mcpi_dist(loops): count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 return(count)

End-User Guide Techila with Python 30/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The function 'mcpi_dist' is very similar to the algorithm of the locally executable function. The function takes one input argument called 'loops' which will be used to determine the number of iterations performed during the Job. During each iteration, the distance of a randomly generated point from the centre will be calculated.

If the distance is less than one, the point is within the unit circle and the count is incremented by one.

No post-processing activities are performed in the Worker Code, as the results from individual Jobs will be post-processed in the Local Control Code after all Jobs have been completed. Creating the computational project To create the computational Project, change your current working directory in your Python environment to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_mcpi.py') result = run_mcpi(10,1000000) This will create a Project consisting of ten Jobs, each containing 1,000,000 iterations.

The Jobs will be distributed to Workers, where the Monte Carlo routine in the Worker Code is executed. When a Worker finishes the Monte Carlo routine, results are transferred to the Techila Server. After all the Workers have transferred the results to the Techila Server, the results are transferred to the End-Users computer. After the results have been downloaded, the post-processing operations will be performed and the approximated value of Pi will be displayed.

Figure 19. Creating the computational Project. The function will return the approximated value of Pi based on the values generated in computational Jobs.

End-User Guide Techila with Python 31/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

3.2 Streaming & Callback Function Streaming enables individual results generated in computational Jobs to be transferred to your computer as soon as they become available. This is different from the default implementation, where all the results will be transferred in a single package after all of the Jobs have been completed. Callback functions can be used to post process individual results as soon as they have been streamed from the Techila Server to End-User. The callback function is called once for each result file that will be transferred from the Techila Server.

The example presented in this Chapter uses streaming and a callback function. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\Python\features\streaming_callback Streaming in Python Streaming is disabled by default. Streaming can be enabled with the following parameter pair: stream = True Callback Functions in Python A callback function can be defined by using the following parameter pair: callback='' The notation will need to be replaced with the name of the function you wish to use. For example, the following syntax would set the function 'callbackfun' as the callback function.

callback = callbackfun The callback function will then be called every time a new Job result has been streamed from the Techila Server to End-User. The callback function will automatically receive one input argument, which will contain the result returned from the function that was executed on the Worker. Values returned by the callback function will be stored as elements in the list that will be returned by the peach-function. The implementation of the Streaming and Callback features will be demonstrated using the Monte Carlo Pi method. In the distributed version of the program, Job results will be streamed as soon as they have become available.

The callback function is used to print the approximated value of Pi each time a new result has been received.

End-User Guide Techila with Python 32/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Local Control Code The Python script containing the Local Control Code for this example is shown below. import techila def callbackfun(jobresult): global total_jobs global total_loops global total_count total_jobs = total_jobs + 1 total_loops = total_loops + int(jobresult.get('loops')) total_count = total_count + int(jobresult.get('count')) result = 4 * float(total_count) / total_loops print 'Number of results included:', total_jobs, 'Estimated value of Pi:', result return(jobresult) def run_streaming(jobs, loops): global total_jobs global total_loops global total_count total_jobs = 0 total_loops = 0 total_count = 0 result = techila.peach(funcname = 'mcpi_dist', params = [loops], files = ['mcpi_dist.py'], peachvector = range(1, jobs + 1), stream = True, callback = callbackfun, ) return(result) The Local Control Code used in this example contains of two functions, 'run_streaming' and 'callbackfun'.

The 'run_streaming' function contains the peach-function call, which will be used to distribute the computations using the peach-function. The syntax of the peach-function defines that individual Job results should be streamed ('stream = True') and that the results should be processed by using a callback function ('callback=callbackfun'). The function 'callbackfun' is the callback function that will be executed every time a new Job result is streamed from the Techila Server to your computer. This function is used to combine the individual results and to display the approximation result each time a new Job result has been received.

The input argument of the callback function ('jobresult') will be a 'dict' object, which is returned by the code that is executed on the Workers. In this example, the object will contain the number of points inside the unitary circle ('count') and the number of iterations performed in the Job ('loops'). The values returned from Jobs will be added together and stored in the global variables (variables starting with 'total_') in order to preserve the values between function calls.

End-User Guide Techila with Python 33/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Each time a new Job result has been processed, the current value of the approximation will be displayed. The callback function will return the individual Job results in the same format as they were received.

These returned values will be stored as a list element in the list returned by the peach-function. Worker Code The Worker Code used in this example is shown below. import random def mcpi_dist(loops): count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 return({'count': count, 'loops': loops}) The code is similar to the basic implementation introduced in Chapter 3.1. The differentiating factor is that the function returns a 'dict' object, which will contain the number of iterations performed in the Job ('loops') and the number of points within the unitary circle ('count').

This object will be automatically passed as an input argument to the callback function 'callbackfun' defined in the Local Control Code. Creating the computational project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_streaming.py') result = run_streaming(20,1000000) This will create a computational Project consisting of 20 Jobs, each Job performing a Monte Carlo routine that consists of 1,000,000 iterations. Results will be streamed from the Techila Server to End-User as they are completed and the approximated value continuously as more results are streamed. This is illustrated below.

End-User Guide Techila with Python 34/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 20. Creating the computational Project. After the Project has been completed, individual results returned from Jobs can be examined by indexing the 'result' list as illustrated in the figure.

End-User Guide Techila with Python 35/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.3 Job Input Files Job Input Files can be used in scenarios, where individual Jobs only require access to some files in a larger dataset. Job-specific input files will be stored in a Job Input Bundle and will be transferred to the Techila Server.

Techila Server will transfer files from the Bundle to the Workers requiring them. These files will be stored on the Worker in the temporary working directory for the duration of the Job. The files will be removed from the Worker as soon as the Job has completed.

The material used in this example is located in the following folder in the Techila SDK:  techila\examples\Python\features\job_input_files Job Input Files in Python The names of Job-specific input files that you wish to transfer to Workers are defined by using the 'jobinputfiles' parameter. This parameter is a dictionary object which has the following keywords:  The datafiles keyword is used to define a list of files that should be transferred to Workers.  The filenames keyword is used to define the name of the file on the Workers  The datadir keyword can be used to specify the path where the files are located on your computer.

If not defined, files will be accessed from the current working directory. An example of a 'jobinputfiles' parameter definition is shown below: jobinputfiles = {'datafiles [ ' file1', 'file2'], 'filenames [ ' workername']} The syntax shown above would transfer file 'file1' to Job #1 and file 'file2' to Job #2. Both files would be renamed to 'workername' after they have been transferred to Workers. Note! When using Job-specific input files, the number of entries in the 'datafiles' parameter must be equal to the number of Jobs in the Project.

If the files you wish to transfer are not located in the current working directory, the path of the file needs to be specified. For example, the following syntax would transfer 'file1' from the directory 'C:\temp'. jobinputfiles = {'datafiles [ ' file1'], 'datadir [ ' C:\\temp\\'], 'filenames [ ' workername']} The use of Job input Files is illustrated using four text files. Each of the text files contains one row of numbers, which will be summed and the value of the summation will be returned as the result. The computational work performed in this example is trivial and is only intended to illustrate the mechanism of using Job-Specific Input Files.

Local Control Code The Local Control Code for creating a project that uses Job-Specific Input Files is shown below. import techila def run_inputfiles(): jobs = 4 result = techila.peach(funcname = 'inputfiles_dist',

End-User Guide Techila with Python 36/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. files = ['inputfiles_dist.py'], jobs=jobs, jobinputfiles = { 'datafiles' : [ 'input1.txt', 'input2.txt', 'input3.txt', 'input4.txt' ], 'filenames [ ' input.txt'] }, ) return result The peach-function syntax used in the example will create a Project consisting of four Jobs. Each of the Jobs will receive one of input files specified in the 'jobinputfiles' parameter. File 'input1.txt' will be transferred to with Job #1, file 'input2.txt' is transferred with Job #2 and so on. Note that the number of entries in the list is equal to the number Jobs in the Project.

After the files have been transferred to Workers, they will be copied to the same temporary working directory with the executable code. Each file will also be renamed to 'input.txt' during this process. Worker Code Worker Code used to perform operations on the Job-specific input files is shown below. import csv def inputfiles_dist(): data = list(csv.reader(open('input.txt', 'rb'), delimiter ) ) row_int = map(int,data[0]) sum_row_int=sum(row_int) return(sum_row_int) In this example, all Jobs will access the input files by using the file name 'input.txt' which was defined in the Local Control Code.

These files will be accessed from the same temporary working directory that contains the executable code, meaning no path definitions will be required. Each Job will sum the numbers in the Job-specific input file and return the value of the summation as the result. Creating the computational project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_inputfiles.py') result = run_inputfiles() This will create a Project consisting of four Jobs. The system will automatically assign a Job-specific input file to each Job, according to definition of the jobinputfiles parameter in the Local Control Code. This is illustrated below in Figure 21.

End-User Guide Techila with Python 37/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 21. Transferring Job-specific input files. All files will be transferred to the Techila Server. The Techila Server transfers the requested Job Input File for each job. These files are renamed on the Workers according to the parameters in the Local Control Code. In this example, the files are renamed to 'input.txt' and copied to a temporary working directory on the Workers.

The output generated when creating the computational Project should resemble the one shown below in Figure 22.

Figure 22. Creating the computational Project. The function will return a list containing four elements, one element for each Job in the Project. Each list element will contain the sum of values read from the Job-specific input file.

End-User Guide Techila with Python 38/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. 3.4 Project Detaching When a Project is detached, the peach-function will return immediately after all of the computational data has been transferred to the Server. This means that Python can be used for other purposes while the Project is being computed. Results of a Project can be downloaded later by performing another peach- function call that will link to the Project ID number of the Project that was created earlier. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\Python\Features\detached_project Project detaching in Python Projects can be detached using following parameter: donotwait = True This will cause the peach-function to return immediately after the Project has been created and all computational data transferred to the Techila Server.

The peach-function call will return the Project ID number, which can be used in the download process.

Results can be downloaded by linking the peach-function call to an existing Project ID number using following parameter pair: projectid = The notation should be replaced by the Project ID number of the Project you wish to download the results for. For example, the following syntax would download results of Project 1466. projectid = 1466 It is also possible to download results of a previously completed Project, even when the original Project was not detached with the 'donotwait' parameter. Please note however that results can only be downloaded if they have not been removed from the Techila Server.

Project ID numbers of previously completed Projects can be viewed from the Techila Web Interface.

The following example demonstrates how to detach a Project and download results using the peach- function. Local Control Code The Local Control Code ('run_detached.py') used in this example is shown below. import techila def run_detached(jobs, loops): pid = techila.peach(funcname = 'mcpi_dist', params = [loops], files = ['mcpi_dist.py'], peachvector = range(1, jobs + 1), donotwait = True, ) return(pid)

End-User Guide Techila with Python 39/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. def download_result(pid): results = techila.peach(projectid = pid) points = 0 for res in results: points = points + int(res.get('count')) result = 4 * float(points) / (len(results) * int(results[0].get('loops'))) return(result) The code contains two functions, 'run_detached' and 'download_result'. The 'run_detached' function contains the peach-function call, which will be used to create the computational Project.

The peach-function syntax contains the following parameter that specifies that the Project should be detached after all necessary data has been transferred to the Techila Server. donotwait = True This parameter will also cause the peach-function to return the Project ID number of the Project that was created. This Project ID number will be stored in the variable 'pid' and will be used later when downloading the results for the Project.

After creating the Project, the 'download_result' function can be used to download the results. This function takes one input argument ('pid'), which will be used to link the peach-function call to a previously created Project. After results have been downloaded, the results will be stored in the 'results' list. Each list element will contain the result returned from one Job. The list elements will then be combined and used to calculate an approximated value for Pi. Worker Code The code that is executed on the Workers is shown below. import random def mcpi_dist(loops): count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 return({'count': count, 'loops': loops}) The code used in this example performs the same Monte Carlo routine as in the basic implementation introduced in Chapter 3.1.

The differentiating factor is that the function returns a 'dict' object, which will contain the number of iterations performed in the Job ('loops') and the number of points within the unitary circle ('count').

End-User Guide Techila with Python 40/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The number of iterations is stored in order to preserve information that is required in the post-processing. Storing all variables required in post-processing in the result files means, that the post-processing activities can be performed correctly regardless of when the results are downloaded.

Creating the computational Project and downloading results To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_detached.py') pid = run_detached(10,10000000) This creates a Project consisting of ten Jobs. After all of the computational data has been transferred to the Techila Server, the Project ID number will be returned and stored to the 'pid' variable. This Project ID number can be used to download the results of the Project after all the Jobs have been completed.

The process of creating the Project is illustrated below.

Figure 23. Creating the Project. The function will return the Project ID number. After the Project has been completed, the results can be downloaded from the Techila Server with the 'download_result' function using the syntax shown below. Please note that if you execute the function before the Project has been completed, the peach-function will wait for the Project to be completed. results = download_result(pid) Figure 24. Downloading the results with the 'download_result' function.

End-User Guide Techila with Python 41/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.5 Iterative Projects Creating Projects iteratively is not so much as a feature as it is a technique. Using this type of an approach is typically used in scenarios where output values of previously completed Projects will be used as input values in future Projects. Creating several consecutive Projects can be achieved by for example placing the peach-function call inside a loop structure.

The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\iterative_projects Local Control Code The Local Control Code used to create several, consecutively created projects is shown below. import math import techila def run_iterative(): threshold = 0.00004 jobs = 20 loops = 1e7 total_result = 0 iteration = 1 current_error = math.pi techila.init() while abs(current_error) >= threshold: result = techila.peach( funcname = 'mcpi_dist', params = [loops , iteration], files = ['mcpi_dist.py'], peachvector = range(1, jobs + 1), messages = False, ) if result == None or len(result) == 0: break total_result = total_result + sum(result) approximated_pi = float(total_result) * 4 / (loops * jobs * iteration) current_error = approximated_pi - math.pi print 'Amount of error in the approximation , current_error iteration = iteration + 1 print('Error below threshold, no more Projects needed.') techila.uninit() return current_error The Local Control Code used in this example contains three logical steps:  Step 1: Initializing the connection to the Techila Server  Step 2: Iterative Project creation  Step 3: Uninitializing the connection

End-User Guide Techila with Python 42/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. These steps are explained below: Step 1: The connection to the Techila Server will be initialized by calling the 'init' function before performing the peach-function call.

When 'init' is called before the peach-function, the peach- function will not uninitialize the connection after downloading the results. This means that the connection will remain available for subsequent peach-function calls. This also means that when creating several consecutive Projects, performing a separate connection initialization is typically more efficient than performing a new initialization with each peach-function call.

Step 2: The peach-function call used to create the computational Projects is placed inside a loop structure, which is implemented with a 'while' statement. Each computational Project will be used to compute an approximation for the value of Pi. The results generated in the Projects will be combined and compared to the value of Pi ('math.pi') to compute the current error in the approximation. New computational Projects will be created until the error in the approximation is below the predefined threshold value. The amount of error in the approximation will be printed every time a new Project has been completed.

Note that messages have been disabled ('messages = False') in order to provide a more clear illustration of the results received from Projects.

Step 3: After the error in the approximation drops below the threshold, the loop structure will be exited. The connection to the Techila Server will be uninitialized by using the 'uninit' function, which will also remove all temporary files created by the peach-function. The run_iterative-function will return the amount of error in the approximation after all Projects have been completed. Worker Code The algorithm for the Worker Code is shown below. import random def mcpi_dist(loops, jobidx, iteration): random.seed(jobidx * iteration) count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 return count The executable function takes three input arguments: 'loops', 'jobidx' and 'iteration'.

The 'loops' variable is used to define the number of iterations performed in the while-loop. The 'jobidx' and 'iteration' variables are used to specify the seed of the random number generator. This is done in order to ensure that the number of required Projects required stays within a reasonable limit. Apart from the modifications described above, the computational operations performed in the Worker Code are similar as in the basic implementation presented in Chapter 3.1, returning the number of random points that are located within the unitary circle.

End-User Guide Techila with Python 43/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Creating the computation Project To create the computational Projects, change your current working directory (in Python) to the directory that contains the example material relevant to this example.

After having browsed to the correct directory, create the computational Projects using commands: execfile('run_iterative.py') result=run_iterative() Executing the commands shown above will create several, consecutively created Projects each consisting of 20 Jobs. New Projects will be created until the error in approximation is smaller than the threshold value. The error of the approximation will be printed every time a Project has been completed. Figure 25. The error in the approximation will be displayed each time a new Project is completed. New Projects will be created until the error in the approximation drops below the threshold.

End-User Guide Techila with Python 44/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.6 Data Bundles Data Bundles can be used to efficiently transfer and manage large amounts of data in computational Projects. After being created, Data Bundles will be stored on the Techila Server from where they will be automatically used in future Projects, assuming that the content of the Data Bundle does not change.

If the content of the Data Bundle changes (e.g. new files added, existing files removed or the content of existing files modified), a new Data Bundle will be automatically created and transferred to the Techila Server. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\data_bundle Data Bundles in Python Files can be stored in Data Bundles by using the 'databundle' parameter. For example, the syntax shown below would store files called 'file1' and 'file2' in to a Data Bundle databundles { ' datafiles [ ' file1','file2']}] The listed files will be transferred to Workers, where they will be copied to the same temporary working directory with the executable code.

An expiration period can be defined for the Data Bundle, which will determine the time period how long an unused Bundle will be stored on a Worker. If a value is not defined, the default expiration periods will be used. For example, an expiration period of 30 minutes can be defined with the following syntax: databundles { ' datafiles [ ' file1','file2'], 'parameters { ' ExpirationPeriod' : '30 m'}}] Several Data Bundles can be created by defining additional list entries for the 'databundles' parameter. For example, the syntax shown below would create two Data Bundles. The first Data Bundle contains files 'file1' and 'file2' and the second Data Bundle contains files 'file3' and 'file4'.

databundles { ' datafiles [ ' file1','file2']}, {'datafiles [ ' file3','file4']}] By default, the files listed in the 'datafiles' parameter will be read from the current working directory. If the files are not located in the current working directory, the path of the files can be defined with the 'datadir' parameter. For example, the syntax shown below would read files 'file1' and 'file2' from the path 'C:\temp'. Files 'file3' and 'file4' would be read from the current working directory. databundles { ' datafiles [ ' file1','file2'], 'datadir' : 'C:\\temp'}, {'datafiles [ ' file3','file4']}] This example illustrates how to transfer data files using two Data Bundles Local Control Code The Local Control Code used in this example is shown below.

import techila def run_databundle():

End-User Guide Techila with Python 45/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. result = techila.peach( funcname = 'databundle_dist', files = ['databundle_dist.py'], jobs = 1, databundles # Define dadatabundles { # Data Bundle #1 'datadir . / storage/', 'datafiles' : [ 'file1_bundle1', 'file2_bundle1', ], 'parameters' : { 'ExpirationPeriod' : '60 m', } }, { # Data Bundle #2 'datafiles' : [ 'file1_bundle2', 'file2_bundle2', ], 'parameters' : { 'ExpirationPeriod' : '30 m', } } ] ) return(result) The Local Control Code used in this example will create two Data Bundles.

Files 'file1_bundle1' and 'file2_bundle1' will be read from a folder called 'storage' and stored in the first Data Bundle. Files 'file1_bundle2' and 'file2_bundle2' will be read from the current working directory and stored in the second Data Bundle. Expiration periods of the Data Bundles will be set to 60 minutes for the first Bundle and 30 minutes for the second Bundle.

Worker Code The Worker Code used in this example is shown below. import csv def databundle_dist(): a = list(csv.reader(open('file1_bundle1', 'rb'), delimiter ) ) b = list(csv.reader(open('file2_bundle1', 'rb'), delimiter ) ) c = list(csv.reader(open('file1_bundle2', 'rb'), delimiter ) ) d = list(csv.reader(open('file2_bundle2', 'rb'), delimiter ) ) a_int = map(int,a[0]) b_int = map(int,b[0]) c_int = map(int,c[0]) d_int = map(int,d[0]) return([a_int,b_int,c_int,d_int])

End-User Guide Techila with Python 46/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The Worker Code contains instruction for reading each of the files included in the Data Bundles. The files in the Data Bundles will be copied to the same temporary working directory as the executable Python code, meaning the files can be accessed without any additional path definitions. After reading the files, the executable function will return a list containing the values read from the files.

The value returned from the Job will be eventually returned by the peach-function defined in the Local Control Code.

Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_databundle.py') result = run_databundle() This creates a Project consisting of one (1) Job. Two Data Bundles will be created and transferred to the Techila Server, from where they will be transferred to the Worker. Please note that if you execute the Local Control Code several times, the Data Bundles will only be created with the first Project.

Any subsequent Projects will use the Data Bundles stored on the Techila Server.

Figure 26. Creating the computational Project. In this example, the Project will contain one Job, which will consist of reading all files that were transferred to the Worker. The Job will return the values in a list, which will in turn also returned by the 'run_databundle' function.

End-User Guide Techila with Python 47/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. 3.7 Function handle Function handles (pointers to another function) can be used as values for the 'funcname' parameter. When passing a function handle as a value to the 'funcname' parameter, no separate Python script for the Worker Code will be required. This means that the 'files' parameter does not need to be defined. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\function_handle Function handles in Python Function handles can be used for defining the executable function using the syntax shown below: funcname = The notation should be replaced with the function you wish to execute on Workers.

For example, the following syntax would execute a function called 'testfun' on all Workers. funcname = testfun Please note that the function needs to be defined, before it can be used as the value of the 'funcname' parameter. Also note that when referring to a function handle, quotation marks are not used. If the function performs calls to additional functions, these functions will not be available on the Worker. In order to make multiple functions available, please use the 'files' parameter. This example illustrates how to use the 'funcname' parameter to execute a function that is defined in the local Python environment on the Workers.

Local Control Code The Local Control Code used in this example is shown below. import techila def mcpi_dist(loops): import random count = 0 i = 0 while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 i = i + 1 return count def run_funchandle(jobcount, loops): result = techila.peach(funcname = mcpi_dist, params = [loops], jobs = jobcount, ) result = 4 * float(sum(result ( jobs * loops) print 'The approximated value of Pi is:', result return result

End-User Guide Techila with Python 48/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The Local Control Code shown above defines two functions; 'mcpi_dist' and 'run_funchandle'. The run_funchandle-function contains the peach-function call that will be used to create the computational Project. The 'funcname' parameter of the peach-function call refers to the mcpi_dist- function and is entered without quotation marks.

The mcpi_dist-function contains the executable code that will be executed on Workers during the computational Jobs. This function will be defined when the Local Control Code is evaluated using the 'execfile' function. Please note that the 'files' parameter is not used, meaning that no Python scripts will be sourced during the preliminary stages of a computational Job. Creating the computational Project To create the computational Project, change your current working directory in your Python environment to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_funchandle.py') result = run_funchandle(10,1000000) This will create a Project consisting of ten Jobs, each Job performing 1,000,000 iterations of the Monte Carlo routine defined in the mcpi_dist function.

The values returned from the Jobs will be used to calculate an approximate value for Pi.

End-User Guide Techila with Python 49/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.8 File Handler The file handler is a function that can be used to process additional output files after they have been transferred to the End-Users computer from the Techila Server.

The file handler function can be used for example to transfer the files to suitable directories or to perform other post-processing activities. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\file_handler Additional output files and the file handler function In order to return additional output files generated on Workers, the output files need to be defined in the peach-function syntax by using the 'outputfiles' parameter. For example, the following syntax would transfer an output file called 'output_file1' from the Workers to your computer.

outputfiles = ['output_file1'] Several files can be transferred from Workers by defining the names of the files as list elements. For example, the following syntax would specify that two files called 'output_file1' and 'output_file2' should be returned from Workers.

outputfiles = ['output_file1','output_file2'] If the names of the output files that are generated on Workers are different for each Job, regexp notations can be used to return all files matching the specified filter. For example, the following syntax would return all files starting with the string 'output_file' from the Workers. outputfiles = ['output_file.*;regex=1'] Each output file that is transferred to your computer can be processed by a file handler function, which is defined in the Local Control Code. The file handler function will be called once for each transferred file and takes one input argument, which will automatically contain the path and name of the file.

The name of the function that will be called for each output file is defined with the 'filehandler' parameter. For example, the following syntax specifies that a function called 'filehandler_func' should be used as the file handler function.

filehandler=filehandler_func This example illustrates how to process additional output files generated during computational Jobs by using a file handler function

End-User Guide Techila with Python 50/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

Local Control Code The Local Control Code used in this example is shown below. import techila import os import shutil def filehandler_func(file): print file f = open(file, 'r') line = f.readline() print line f.close() filename=os.path.basename(file) shutil.copy(file, os.getcwd()) def run_filehandler(): jobs=5 result = techila.peach(funcname = 'worker_dist', files = 'worker_dist.py', params ] , peachvector = range(1, jobs+1), outputfiles = ['output_file.*;regex=1'], filehandler = filehandler_func, ) In the peach-function syntax shown above, regexp notations have been used to specify all files starting with the string 'output_file' as output files.

These files will be transferred from the Workers to your computer. The regexp notation is required, because each of the output files will have a different name. After the files have been transferred to your computer, the 'filehandler_func' function will be used to process each of the output files. The 'filehandler_func' will be called once for each output file. Each time the function is called, the value of the input argument (variable 'file') will be automatically replaced with the name and path of output file. In this example, the file handler function will be used to print the path and name of the file and to copy the file to the current working directory.

Worker Code The Worker Code used in this example is shown below. def worker_dist(jobidx): sample1 = 'This file was generated in Job + str(jobidx) f = open('output_file' + str(jobidx),'w') f.write(sample1) f.close Each Job in the computational Project will generate one output file. The name of the output file will include the value of the input argument 'jobidx', meaning the name of the file will be different for each Job. Job #1 will generate a file called 'output_file1', Job #2 will generate a file called 'output_file2' and

End-User Guide Techila with Python 51/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. so on. The names of all generated output files will match the regexp filter defined in the Local Control Code, meaning all files will be transferred to the End-Users computer.

Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example. After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_filehandler.py') run_filehandler() This creates a Project consisting of five Jobs. Each Job will generate one additional output file, which will be processed by the file handler function. The output generated by the commands should resemble the one shown below.

Figure 27. Creating the computational Project.

The file handler function will display the temporary locations and content of each output file. The files will be copied to the current working directory as can be seen from the output generated by the 'listdir' function.

End-User Guide Techila with Python 52/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.9 Snapshots Snapshotting is a mechanism where intermediate results of computations are stored in snapshot files and transferred to the Techila Server at regular intervals.

Snapshotting can be used to improve the fault tolerance of computations and to reduce the amount of computational time lost due to interruptions. Snapshotting is performed by storing the state of the computation (i.e. variables required to resume computations) at regular intervals in snapshot files on the Worker. The snapshot files will then be automatically transferred to the Techila Server at regular intervals from the Workers. If an interruption should occur, these snapshot files will be transferred to other available Workers, where the computational process can be resumed by using the intermediate results stored in the snapshot file.

The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\snapshot Snapshots in Python Snapshotting with default values can be enabled with the following parameter pair: snapshot=True The default values will define the following snapshot parameters:  The name of the snapshot file ('snapshot.dat')  The transfer interval of the snapshot file (30 minutes) Variables that you wish to store in the snapshot file will need to be declared global in the code that is executed on Workers. After declaring the variables global, they can be saved by using the 'save_snapshot' function in the Worker Code.

For example, the following syntax would store the variables 'var1' and 'var2' in the snapshot file.

save_snapshot(var1,var2) The variables will be stored in the snapshot file ('snapshot.dat'), which will be transferred to the Techila Server after preconfigured time intervals. This file will be automatically transferred to a Worker if the computational Job becomes interrupted and needs to be assigned to another Worker. Variables stored in a snapshot file can be loaded by using the 'load_snapshot' function in the Worker Code. For example, the following command loads all variables stored in the snapshot file. load_snapshot() The default snapshot transfer interval in Python is 30 minutes. The snapshot transfer interval can be modified with the 'snapshotinterval' parameter.

For example, the syntax shown below will set the transfer interval to five (5) minutes.

snapshotinterval=5 This example demonstrates how to store and load variables into and from snapshot files. Local Control Code The Local Control Code used in this example is shown below.

End-User Guide Techila with Python 53/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners.

import techila def run_snapshot(jobs, loops): result = techila.peach( funcname = 'snapshot_dist', params = [loops], files = ['snapshot_dist.py'], peachvector = range(1, jobs + 1), snapshot = True, ) result = 4 * float(sum(result ( jobs * loops) print 'The approximated value of Pi is:', result return(result) Snapshots are enabled with the following parameter pair in the Local Control Code: snapshot=True No other modifications are required to enable snapshotting with the default snapshot parameters. Apart from the parameter pair used to enable snapshotting, the structure of the Local Control Code is similar to the basic implementation as shown in Chapter 3.1 Worker Code The Worker Code used in this example is shown below.

import random import os import cPickle import sys def snapshot_dist(loops): global count global i count = 0 i = 0 load_snapshot() while i < loops: if pow(pow(random.random(), 2) + pow(random.random(), 2), 0.5) < 1: count = count + 1 if i > 0 and i % 1e7 == 0: save_snapshot('i', 'count') i = i + 1 return(count)

End-User Guide Techila with Python 54/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. During the initial steps in the executable function the 'count' and 'i' variables are declared global and initialized. The initialization values will be used in situations where a snapshot file cannot be found (e.g.

when starting the Job). If a snapshot file exists, it will indicate that the Job is being resumed after an interruption. In this case, the content of the snapshot file will be used to override the initialized values. This will be performed using the 'load_snapshot' function, which automatically loads variables stored in the snapshot file to the Python environment. After loading the snapshot file, computations will be resumed from the last iteration value ('i') stored in the snapshot file.

Intermediate results will be stored in the snapshot file by calling the 'save_snapshot' function every1e7th iteration. The variables stored in the snapshot file are 'i' and 'count'. The parameter 'i' will contain the number of iterations performed when the snapshot generation occurred. The parameter 'count' will contain the intermediate results. Creating the computational Project To create the computational Project, change your current working directory (in Python) to the directory that contains the example material for this example.

After having navigated to the correct directory, create the computational Project using commands shown below: execfile('run_snapshot.py') result=run_snapshot(10,1e8) This creates a Project consisting of 10 Jobs, each Job performing 1e8 iterations.

Intermediate results will be saved at every 1e7th iteration. Snapshot files will be transferred approximately every 30 minutes from the Worker to Techila Server. If a Job is migrated to a new Worker while the Job is being computed, the latest available snapshot file will be automatically transferred from the Techila Server to the new Worker. Snapshot data can also be viewed and downloaded by using the Techila Web Interface. Instructions for this can be found in the Techila Web Interface End-User Guide.

Note that when using the syntax shown above to run the example, the execution time of single Job is relatively short. This might result in the Job being completed before a snapshot file will be transferred to the Techila Server. If snapshot data is not visible in the Techila Web Interface, consider increasing the amount of iterations to increase the execution time of a Job. You can also experiment by specifying a shorter snapshot interval with the 'snapshotinterval' parameter as explained earlier in this Chapter.

End-User Guide Techila with Python 55/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.10 Using custom Python packages in computational Projects Custom Python packages can be made available on Workers by either using the 'packages' parameter of the peach-function or by using the bundleit-function to create the Bundle. This Chapter discusses using the bundleit-function in more detail.

If you wish to see how the 'packages' parameter is used to create a Bundle, please refer to the file 'run_packagetest2.py'. This file is located in the same directory that contains the example material that uses the bundleit-function. When using the bundleit-function, Python packages that are not included in the standard Python distribution can be made available on Workers by performing the steps listed below. Step 1: Creating a Bundle from the Python package using the 'bundleit' function Step 2: Importing the Bundle in to a computational Project by using the 'imports' parameter These steps are described below.

Step 1: Creating the Python Package Bundle Python packages that are installed on your computer can be stored in Python Package Bundles by using the 'bundleit' function, which is included in the 'techila' package. Bundles that are created with this function will be automatically transferred to the Techila Server and can be imported in to computational Projects. The general syntax of the 'bundleit' function is shown below: bundleit(package, version = None, all_platforms = False, expiration = '7 d', bundlename = None, sdkroot = None, initfile = None, password = None): The input parameters of the function are explained below: The 'package' parameter defines name of the installed Python package you wish to store in the Bundle and transfer to the Techila Server.

For example, the following syntax could be used to transfer a package named 'techilatest' to the Techila Server.

>>> import techila >>> bundlename=techila.bundleit(package='techilatest') >>> bundlename 'demouser.Python.v272.package.techilatest.1.0.0' The 'bundleit' function will return the name of the Bundle that was created. In the example shown above, the name will be stored in the variable 'bundlename'. The Bundle name returned by the 'bundleit' function can be used to import the Bundle (Step 2) to a computational project, so it is typically convenient to store the value in a variable for future use. The general naming convention of Bundles created with the 'bundleit' function is illustrated below.

End-User Guide Techila with Python 56/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 28. The naming convention of Bundles created by the bundleit-function. The 'version' parameter can be used to modify the name of the Bundle that will be created. Modifying the name of the Bundle will typically be required if you wish to re-create a Bundle from a Python package that has already been transferred to the Techila Server.

This is because all Bundles must have unique names and Bundles with a non-unique name will not be accepted by the Techila Server. The example below shows how to modify the name of a Bundle with the 'version' parameter. >>> import techila >>> bundlename=techila.bundleit(package='techilatest',version='1.0.1') >>> bundlename 'demouser.Python.v272.package.techilatest.1.0.1' The 'all_platforms' parameter can be used to specify that the Bundle should be offered to all Workers, regardless of the operating system of the Worker. By default, Bundles will only be offered to Workers that have the same operating system type (Linux or Windows) as the one used by the End-User when creating the Bundle.

To allow all Workers to use the Bundle, specify the parameter as shown below: all_platforms=True The 'expiration'parameter can be used to specify a different expiration period (default value 7 days) for the Bundle. This expiration period defines how long the Bundle can remain unused on the Worker, before it will be marked for removal. For example, the syntax shown below could be used to set an expiration period of 30 days.

expiration = '30 d' The 'sdkroot' parameter can be used to specify the path of the 'techila' directory on your computer. If you have set the environment variable 'TECHILA_SDKROOT', using this parameter will not be required. The 'initfile' parameter can be used to specify the path of the 'techila_settings.ini' file if the file cannot be found from any of the default locations (for example the 'techila' directory or the user home directory).

The 'password' parameter can be used to specify the password of the End-User Keystore. When specified, the password will not be prompted when creating the Bundle.

Step 2: Importing the Bundle Bundles containing Python packages can be imported to computational Projects by using the 'imports' parameter of the peach-function. The general syntax for defining the parameter is shown below. imports=['']

End-User Guide Techila with Python 57/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. For example, the following syntax could be used to import the Bundle that was used as an example in Step 1 earlier in this Chapter: imports=['demouser.Python.v272.package.techilatest.1.0.0'] Bundles listed in the 'imports' parameter will be transferred to all Workers that participate in the Project.

After being imported, the packages stored in the Bundles can be used normally in the Worker Code.

The following example illustrates how to store a Python package in a Bundle and import the Bundle in a computational Project. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\custom_package Local Control Code and the 'techilatest' package In this example, a package called 'techilatest' will be installed on the End-Users computer. This package will then be transferred to the Techila Server by using the 'bundleit' function. The package is located in the same folder that contains the example material for this example and contains one module called 'functions.py'.

The content of the 'function.py' file is shown below. Instructions for installing the package can be found at the end of this Chapter. def summation(a,b): return(a+b) def multiplication(a,b): return(a*b) def subtraction(a,b): return(a-b) def division(a,b): af=float(a) bf=float(b) return(af/bf) The functions shown above can be used to perform basic arithmetic operations. Each function takes two input arguments. These functions will be called during the computational Jobs. The input arguments will be defined in the Local Control Code shown below.

The Local Control Code (in file 'run_packagetest.py') used in this example is shown below.

import techila def create_package(): bundlename = techila.bundleit('techilatest', all_platforms = True) print 'package bundlename is \'%s % bundlename return(bundlename)

End-User Guide Techila with Python 58/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. def run_packagetest(bundlename): results = techila.peach(funcname = 'packagetest_dist', params ] , files = ['packagetest_dist.py'], imports = [bundlename], peachvector = [1, 2, 4, 8, 16], ) for row in results: print row The script contains two functions, which will be used to perform the following operations: The function 'create_package' contains the bundleit-function call, which will be used to create a Bundle from the 'techilatest' package.

The function will return the name of the Bundle ('bundlename') that was created, which can be used to import the Bundle to the computational Project. The function 'run_packagetest' contains the peach-function call, which will be used to create the computational Project. The function takes one input argument ('bundlename') which will be used to define which Bundle should be imported to the Project ('imports = [bundlename]'). The peach-function syntax also defines two input arguments ('params ) , which will be passed to the executable function. These input arguments will eventually be passed to the functions in the 'techilatest' package.

Worker Code The Worker Code used in this example is shown below. import techilatest.functions as test def packagetest_dist(a,b): res1 = test.summation(a,b) res2 = test.multiplication(a,b) res3 = test.subtraction(a,b) res4 = test.division(a,b) return([res1,res2,res3,res4]) The first line in the Worker Code will load the 'functions' module from the 'techilatest' package. The import can be performed because the Bundle containing the package has been transferred to all participating Workers. After importing the module, the functions can be accessed during the computational Job.

Installing the 'techilatest' package and creating the computational Project To install the 'techilatest' package, launch a command prompt / terminal and change your current working directory to the directory that contains the example material for this example.

Install the package using command: python setup.py install Executing the command should generate a similar output as shown below (paths may differ).

End-User Guide Techila with Python 59/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. C:\techila\examples\python\Features\custom_package>python setup.py install running install running build running build_py copying techilatest\functions.py -> build\lib\techilatest copying techilatest\__init__.py -> build\lib\techilatest running install_lib copying build\lib\techilatest\functions.py -> C:\Python27\Lib\site- packages\techilatest byte-compiling C:\Python27\Lib\site-packages\techilatest\functions.py to functions.pyc running install_egg_info Removing C:\Python27\Lib\site-packages\techilatest-1.0-py2.7.egg-info Writing C:\Python27\Lib\site-packages\techilatest-1.0-py2.7.egg-info After installing the package, launch Python and follow the instructions below to run the example.

Evaluate the Python script containing the Local Control Code using command: execfile('run_packagetest.py') Create a Bundle from the package 'techilatest' using command: bundlename = create_package() Create a computational Project using command: result = run_packagetest(bundlename) This will create a Project containing five Jobs. The Bundle containing the 'techilatest' package will be transferred to each participating Worker and the package will be available on these Workers. The output generated by the commands should resemble the one shown below.

End-User Guide Techila with Python 60/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 29. Creating a Bundle from the 'techilatest' package and creating the Project. After the Project has been completed, the values generated in the Jobs will be displayed.

Each displayed row contains the values returned from one Job and contains the values returned from the functions in the 'techilatest' package.

End-User Guide Techila with Python 61/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 3.11 Precompiled binaries The Python peach-function can also be used to execute precompiled binaries on Workers. Please note that when executing precompiled binaries on Workers, the syntax for defining input arguments and output files will be different.

This is because the wrapper 'peachclient.py' will not be used. The material used in this example is located in the following folder in the Techila SDK:  techila\examples\python\Features\precompiled_binary Precompiled binaries in Python When executing precompiled binaries during a computational Project, the following peach-function parameter needs to be defined.

executable = True In addition, as precompiled binaries typically do not require Python runtime components on Workers, it is recommended that you also disable Python Runtime Bundles imports with the following parameter: python_required = False In addition to these two parameters, the names of the executable binaries will need to be defined by using the 'binaries' parameter. The general syntax for defining the parameter is shown below. binaries { ' file ' ] The notation should be replaced with path and name of the executable binary you wish to transfer and execute on Workers. For example, the following syntax could be used to transfer file 'mcpi.exe' from the directory 'C:\temp' to Workers.

binaries { ' file': 'C:\\temp\\mcpi.exe'] If the executable is operating system specific (e.g. a Windows executable), the 'osname' parameter can be used to define the operating system of Workers that are allowed to execute the binary. For example, the following syntax defines that the executable should only be executed on Workers with a Windows operating system.

binaries { ' file': 'C:\\temp\\mcpi.exe', 'osname': 'Windows'}] Multiple operating system specific binaries can be executed in the same Project by listing the executables and the applicable operating systems. For example, the following syntax would execute binary 'mcpi.exe' on Workers with a Windows operating system and binary 'mcpi' on Workers with a Linux operating system. binaries { ' file': 'mcpi', 'osname': 'Linux'}, {'file': 'mcpi.exe', 'osname': 'Windows'}] If the executable is processor architecture specific (i.e. only for 32-bit or 64-bit), the processor architecture can be specified with the 'processor' parameter.

For example, the following syntax could be used Input arguments to the executable binary can be passed by using 'params' parameter. For example the following syntax could be used to define three input arguments for the executable binary. Example:

End-User Guide Techila with Python 62/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. binaries { ' file': 'mcpi.exe', 'osname': 'Windows'}] params = '1 10000 example.file' With the example shown above, each Job in Project would fundamentally consist of the following command line instruction being executed on the Worker.

mcpi.exe 1 10000 example.file Input arguments that have a different value for each Job, can be passed by using the %P(jobidx)notation. The value of this notation will be replaced with '1' for Job #1, '2' for Job #2 and so on. These values will be automatically generated on the Techila Server. Example: binaries { ' file': 'mcpi.exe', 'osname': 'Windows'}], params = '%P(jobidx) 10000 example.file' With the example shown above, the following commands line instructions would be executed Workers. Job Executable Argument #1 Argument #2 Argument #3 Job #1 mcpi.exe 1 10000 example.file Job #2 mcpi.exe 2 10000 example.file Job #3 mcpi.exe 3 10000 example.file Table 1.

Example commands in a Project consisting of 3 Jobs. Each Job will execute the 'mcpi.exe' executable. The value of argument #1 will be different for each Job.

The %P() notation can also be used to refer to values of variables that have been defined in the 'project_parameters' parameter. The general syntax for defining a parameter is shown below. project_parameters : } The notation should be replaced by the string that you want to use when referencing the value (i.e. %P(). The notation should be replaced by the value that you wish to use as an input argument. Example binaries { ' file': 'mcpi.exe', 'osname': 'Windows'}], params = '%P(jobidx) %P(loops) example.file', project_parameters = {'loops':10000} Job Executable Argument #1 Argument #2 Argument #3 Job #1 mcpi.exe 1 10000 example.file Job #2 mcpi.exe 2 10000 example.file Job #3 mcpi.exe 3 10000 example.file Table 2.

Example commands in a Project consisting of 3 Jobs. The value of argument #2 will be replaced by the value defined in the 'project_parameters' parameter. Output files that should be transferred from the Workers to your computer can be listed with the 'outputfiles' parameter. The general syntax for defining a file that should be returned is shown below outputfiles=[';file='], The notation should be replaced with a string you wish to use when referencing to the output file (i.e. %O()). The notation should be replaced with the name of the file you wish to transfer from the Workers.

End-User Guide Techila with Python 63/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Example: binaries { ' file': 'mcpi.exe', 'osname': 'Windows'}], params = '%P(jobidx) %P(loops) %O(output)', project_parameters = {'loops':10000}, outputfiles=['output;file=example.file'], Job Executable Argument #1 Argument #2 Argument #3 Job #1 mcpi.exe 1 10000 example.file Job #2 mcpi.exe 2 10000 example.file Job #3 mcpi.exe 3 10000 example.file Table 3.

Example commands in a Project consisting of 3 Jobs. The value of argument #3 will be replaced by the value defined in the 'outputfiles' parameter. If the executable binary executed on Workers generates an output file having a different name for each Job, regexp notations can be used to specify wildcards for the output file name. For example, the following syntax could be used to return all files starting with the string 'output_file' from the Workers. outputfiles=['output;file=example.file;regex=1'] The mechanism for distributing binaries is demonstrated with a precompiled version of the Monte Carlo Pi written in C.

The binary is provided for two platforms:  32-bit Windows (file 'mcpi.exe')  32-bit Linux (file 'mcpi') The generic syntax for executing the binary is: mcpi The input arguments passed to the executable are explained below:  , initializes the random number generator seed  , number of iterations performed  , the name of the output file that will be generated The binaries can be executed locally by following the steps below: 1. Launch a command prompt / terminal 2. Execute the program using command: mcpi 1 1000000 data This executes the binary and performs 1,000,000 iterations of the Monte Carlo Pi routine.

Executing the program with the syntax shown above will create file called 'data', which will contain two values. The first value will be the number of points that were located inside the unit circle. The second value will be the total number of iterations performed. Values will be stored as plain text and the file can be viewed with a text editor.

End-User Guide Techila with Python 64/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Local Control Code The Local Control Code used to create the Project is shown below. import techila def fh(file): f = open(file, 'r') l = f.readline() f.close() values = l.split() in_points = int(values[0]) all_points = int(values[1]) print 'Path of the output file:\n', file print 'Values read from the output file:\n', l return (in_points,all_points) def run_binaries(jobcount=20,loops=100000000): results = techila.peach( executable = True, python_required = False, funcname = 'Precompiled binary', binaries { ' file': 'mcpi', 'osname': 'Linux'}, {'file': 'mcpi.exe', 'osname': 'Windows'}], project_parameters = {'loops':loops}, outputfiles=['output;file=output.data'], params = '%P(jobidx) %P(loops) %O(output)', jobs = jobcount, filehandler = fh, stream = True, ) (in_points,all_points)=map(lambda *x:x ,*results) pivalue= 4*float(sum(in_points))/float(sum(all_points)) print 'Approximated value of Pi ', pivalue return(pivalue) The script shown above defines two functions; 'run_binaries' and 'fh'.

The 'run_binaries' function contains the peach-function call, which will be used to create the computational Project. The parameters of the peach-function are explained below: executable = True, The 'executable' parameter has been set to 'True', indicating that the executable code will be a precompiled binary.

python_required = False

End-User Guide Techila with Python 65/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. The 'python_required' parameter has been set to 'False', indicating that Python Runtime components will not be required on the Worker.

funcname = 'Precompiled binary' As the executable program will be a precompiled binary, the value of the 'funcname' parameter can be defined quite freely. The value of the 'funcname' parameter will only be used to specify the name of the Project (visible in the Techila Web Interface) and for naming the '.state'file that will be generated in to the current working directory when creating the Project. binaries { ' file': 'mcpi', 'osname': 'Linux'}, {'file': 'mcpi.exe', 'osname': 'Windows'}], The 'binaries' parameter defines that the executable 'mcpi.exe' will be executed on Workers with a Windows operating system and the executable 'mcpi' on Workers with a Linux operating system.

These executables will be transferred from the current working directory to the Techila Server when the Project is created.

project_parameters = {'loops':loops}, The 'project_parameters' parameter sets a value for the 'loops' key according to the value of the 'loops' variable. This definition will be required in order to refer to the value using the %P(loops) notation when defining the 'params' parameter. outputfiles=['output;file=output.data'], The 'outputfiles' parameter defines that the file called 'output.data' should be returned from the Workers to the End-Users computer. The syntax also defines the resource 'output', meaning the notation %O(output) can be used to refer to the name of the output file.

params = '%P(jobidx) %P(loops) %O(output)', The 'params' parameter defines three input arguments for the executable binary.

The first input argument (%P(jobidx)) will be replaced by different values for each Job. The value of this notation will be '1' for Job #1, '2' for Job #2 and so on. This value will be used to specify the seed for the random number generator. The second input argument (%P(loops)) will be replaced by the value defined for the 'loops' parameter in 'project_parameters'. The value of this notation will therefore be '100000000' for all Jobs. This value will be used to specify the number of iterations that will be performed in each Job (will define the length of a computational Job).

The third input argument (%O(output)) will be replaced with the value of the resource 'output' defined in the 'outputfiles' parameter. The value of this notation will therefore be 'output.data' for all Jobs. This value will define the name of the file that will be generated by the executable binary. Figure 30 below illustrates how the parameters defined in the Local Control Code will be passed to the executable binary as input arguments.

End-User Guide Techila with Python 66/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Figure 30. Input arguments passed to the executable program will be defined in the Local Control Code by using the 'params' parameter. Values for the 'jobidx' parameter will be generated on the Techila Server.

The 'fh' is the file handler function which will be used to process each of the output files generated during the computational Project. In this example, the function will be used to:  Display the location of the temporary output file after it has been transferred to your computer  Display the values read from the file  Return the values read from the file The values returned by the fh-function will be stored as list elements in the 'results' list that will be returned from the peach-function. After all output files have been processed, the values will be combined and used to calculate an approximated value for pi.

Worker Code Each Worker performs the Monte Carlo routine by executing the 'mcpi.exe' or 'mcpi' binary depending on the operating system of the Worker. The binary is executed according to the input arguments that were defined in the peach-function 'params' parameter. Creating the computational Project The project can be created with the following commands: execfile('run_binaries.py') result = run_binaries() This will create a project consisting of 20 Jobs. During each Job, the 'mcpi' binary will be executed and will perform a Monte Carlo routine that consists of 100,000,000 iterations. After the Monte Carlo routine

End-User Guide Techila with Python 67/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. has been completed, the result of the approximation will be stored a file called 'output.data'. This file will be returned from the Workers to the Techila Server.

As the peach-function syntax enabled result streaming, the results will be streamed from the Techila Server to your computer as soon as they are available.

Each time a new result has been transferred, the file handler function will be executed and used to process the output file. The output generated by the example should resemble the one illustrated below in Figure 31. Figure 31. Creating the computational Project. Output files will be streamed and managed by the file handler function defined in the Local Control Code.