TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

END-USER GUIDE TECHILA WITH PYTHON JANUARY 04, 2013

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. Disclaimer Techila Technologies Ltd. disclaims any and all warranties, express, implied or statutory regarding this document or the use of thereof by you to the full extent permitted by law.

Without limiting the generality of the foregoing, this document provided by Techila Technologies Ltd. in connection therewith are provided 'as-is' and without warranties of any kind, including, without limitation, any warranties of performance or implied warranties of merchantability, fitness for a particular purpose, title and noninfringement. Further, Techila Technologies Ltd. does not make, and has not made, any presentation or warranty that the document is accurate, complete, reliable, current, error-free, or free from harmful information. Limitation of Liability In no event shall Techila Technologies Ltd.

or any of its respective directors, officers, employees, or agents, be liable to you or any other person or entity, under any theory, including without limitation negligence, for damages of any kind arising from or related to the application of this document or any information, content, or materials in or accessible through this document, including, but not limited to, direct, indirect, actual, incidental, punitive, special or consequential damages, lost income, revenue or profits, lost or damaged data, or other commercial or economic loss, that result from your use of, or inability to use, this document, even if any of those persons or entities have been advised of the possibility of such damages or such damages are foreseeable.

Use of this document and copyright No part of this document may be used, reproduced, modified, or transmitted in any form or means without the prior written permission of Techila Technologies. This document and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws. Copyright Techila Technologies 2013. All rights reserved.

TECHILA WITH PYTHON - JANUARY 04, 2013

©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. Table of contents 1 Introduction . . 4 1.1 Installing the techila package . . 5 1.2 Example material . . 5 1.3 Naming convention of the python scripts . . 6 1.4 Python peach-function . . 6 1.5 Process flow in computational Projects . . 7 1.5.1 The peach-function . . 7 1.6 Techila environment variables . . 8 2 Peach tutorial examples . . 9 2.1 Executing a simple Python function on Workers . . 10 2.2 Using input parameters . . 14 2.3 Transferring data files . . 19 2.4 Multiple functions in an Python script . . 23 3 Peach feature examples .

. 26 3.1 Monte Carlo Pi with peach . . 28 3.2 Streaming & Callback Function . . 31 3.3 Job Input Files . . 35 3.4 Project Detaching . . 38 3.5 Iterative Projects . . 41 3.6 Data Bundles . . 44 3.7 Function handle . . 47 3.8 File Handler . . 49 3.9 Snapshots . . 52 3.10 Using custom Python packages in computational Projects . . 55 3.11 Precompiled binaries . . 61

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 4/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1 Introduction This document is intended for Techila End-Users who are using Python as their main development environment. If you are unfamiliar with Techila terminology or the operating principles of the Techila system, please see document "Techila Fundamentals" for more information.

The structure of this document is as follows: Chapter 1 contains important information regarding the installation of required Python packages that enable you to use Techila with Python. This Chapter also contains a brief introduction on the naming convention of the Python-scripts and introduces the peach-function, which is used for distributing computations from Python to the Techila environment.

Chapter 2 contains walkthroughs of simplistic example code samples that use the peach-function. The example material illustrates how to control the core features of the peach-function, including defining input arguments for the executable function, transferring data files to the Workers and calling different functions from the Python script that is evaluated on the Worker. After examining the material in this Chapter you should be able split a simple locally executable program into two pieces of code (Local Control Code and Worker Code), which in turn can be used to perform the computations in the Techila environment.

Chapter 3 contains walkthroughs of several examples that illustrate how to implement different features available in the peach-function. Each subchapter in this Chapter contains a walkthrough of an executable piece of code that illustrates how to implement one or more peach-function features. Each Chapter is named according to the feature that will be the focused on. After examining the material in this Chapter you should be able implement several features available in the peach-function in your own distributed application.

Please note that screenshots in this document are from a Windows 7 operating system.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 5/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 1.1 Installing the techila package The 'techila' Python package is included in the Techila SDK and contains all Techila related Python functions required for creating computational Projects.

Please follow the steps below to install the package. 1. Launch a command prompt \ terminal 2. Change your current working directory to the following directory: cd \techila\lib\python 3. Install the 'techila' package using command: python setup.py install Figure 1.Installing the 'techila' package. 1.2 Example material The Python scripts containing the example material discussed in this document can be found in the Tutorial and Features folders in the Techila SDK. These folders contain subfolders, which contain the actual Python scripts that can be used to run the examples.

TECHILA WITH PYTHON - JANUARY 04, 2013

End-User Guide Techila with Python 6/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. techila examples python Tutorial Features Chapter 2 Chapter 3 Figure 2. The example material discussed in this document can be found in the in the 'python' folder in the Techila SDK 1.3 Naming convention of the python scripts The typical naming convention of Python scripts presented in this document is explained below:  Python scripts ending with '_dist' contain the Worker Code, which will be distributed to the Workers when the Local Control Code is executed.

 Python scripts beginning with 'run_' contain the Local Control Code, which will create the computational Project when executed locally on the End-User’s own computer.  Python scripts beginning with 'local_' contain locally executable code, which does not communicate with the Techila environment. Please note that some Python scripts and functions might be named differently, depending on their role in the computational Project. 1.4 Python peach-function The peach-function provides a simple interface that can be used to distribute Python programs or precompiled binaries. Each peach-function input argument is a named parameter, which refer to a computer language's support for function calls that state the name of each parameter within the function call itself.

A minimalistic Python peach-function syntax typically includes the following parameters:  funcname  params  files  peachvector  datafiles These parameters can be used to define input arguments for the executable function and transfer additional files to the Workers. An example of a peach-function syntax using these parameters is shown below:

End-User Guide Techila with Python 7/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries.

All other trademarks are the property of their respective owners. techila.peach(funcname = 'example_function', # Function executed on Workers params = [var1,var2], # Input arguments to the function files = ['codefile.py # Evaluated at the start of a Job datafiles = ['file1','file2 # Files transferred to Workers peachvector = [2,4,6,8,10] # Peachvector definition ) Examples and more detailed explanations of these parameters can be found in Chapter 2. General information on available peach-function parameters can also be displayed by executing the following commands in Python.

import techila help(techila.peach) 1.5 Process flow in computational Projects When a Project is created with the peach-function, each Job in a computational Project will have a separate Python session. When a Job is started on a Worker, functions and variables are loaded by evaluating the Python script that was defined in the 'files' parameter and by loading the parameters stored in the 'techila_peach_inputdata'file. These parameters will include the parameters defined in the 'params' peach-function parameter.

When a Job is started on a Worker, the 'peachclient.py' script (included in the 'techila' package) is called.

The 'peachclient.py' file is a Python script that acts as a wrapper for the Worker Code and is responsible for calling the executable function, passing input arguments to the function and returning the final computational results. This functionality is hidden from the End-User. The 'peachclient.py' will be used automatically in computational Projects created with peach-function. The 'peachclient.py' wrapper also sets a preliminary seed for the random number generator by using the Python seed() function from the package 'random'. Each Job in a computational Project will receive a unique random number seed based on the current system time and the 'jobidx' parameter, which will be different for each Job.

The preliminary random number seeding can be overridden by calling the seed() function in the Worker Code with the desired random seed.

1.5.1 The peach-function The list below contains some of the Python specific activities that are performed automatically when the peach-function is used to create a computational Project. 1. The peach-function is called locally on the End-Users computer 2. Python scripts listed in the 'files' parameter are transferred to Workers 3. Files listed in the 'datafiles' parameter are transferred to Workers 4. The 'peachclient.py' file is transferred to Workers 5. Input parameters listed in the 'params' parameter are stored in a file called 'techila_peach_inputdata', which is transferred to Workers.

6. The files listed in the 'files' and 'datafiles' parameters and the files 'techila_peach_inputdata'and 'peachclient.py' are copied to the temporary working directory on the Worker 7. The 'peachclient.py' wrapper is called on the Worker. 8. Variables stored in the file 'techila_peach_inputdata' are loaded 9. Files listed in the 'files' parameter are evaluated using the Python 'execfile' command 10. The '' notation is replaced with a 'peachvector' element

End-User Guide Techila with Python 8/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 11. The peachclient calls the function defined in the 'funcname' parameter with the input parameters defined in 'params'.

12. The peachclient saves the result in to a file, which is returned from the Worker to the End-User 13.

The peach-function reads the output file and stores the result in a list element (If a callback function is used, the result of the callback function is returned). 14. The entire list is returned by the peach-function. 1.6 Techila environment variables The table below contains descriptions of environment variables used by Techila. These environment variables can be used to define e.g. the path of the 'techila' directory on your computer or to define the version of Python runtime components used during computational Projects.

Environment variable Description Example value TECHILA_SDKROOT Defines the path of the root directory of the Techila SDK. C:\techila TECHILA_PYTHON_VERSION Defines the Python Runtime Bundle version that will be used in Projects. This parameter can be used e.g. if the Techila environment does not have a Python Runtime Bundle with the same version as your local Python environment. The version of the Runtime Bundle is defined using the syntax . (As returned by command 'sys.version_info' in Python) 273

End-User Guide Techila with Python 9/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd.

All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2 Peach tutorial examples This Chapter contains four minimalistic examples on how to implement and control the core features of the peach-function. The example material discussed in this Chapter, including Python scripts and data files can be found in the subdirectories under the following folder in the Techila SDK:  techila\examples\python\Tutorial Each of the examples contains three pieces of code:  A script containing a locally executable Python function.

This function will be executed locally and will not communicate with the distributed computing environment in any way. This script is provided as reference material to illustrate what modifications are required to execute the computations in the Techila environment.

 A script containing the Local Control Code, which will be executed locally on the End-Users computer. This script contains the peach-function call, which will distribute the computations in the Worker Code to the distributed computing environment  A script containing the Worker Code, which will be executed on the Workers. This script contains the computationally intensive part of the locally executable script. Please note that the example material in this Chapter is only intended to illustrate the core mechanics related to distributing computation with the peach-function. More information on available features can be found in Chapter 3 and by executing the following commands in Python.

import techila help(techila.peach)

End-User Guide Techila with Python 10/67 Author Classification Date Techila Technologies Public January 04, 2013 ©Copyright 2013, Techila Technologies, Ltd. All rights reserved. Techila, Techila Grid, and the Techila logo are either registered trademarks or trademarks of Techila Technologies Ltd in the European Union, in the United States and/or other countries. All other trademarks are the property of their respective owners. 2.1 Executing a simple Python function on Workers This example is intended to provide an introduction on distributed computing using Techila with Python using the peach-function.

The purpose of this example is to:  Demonstrate how to modify a simple, locally executable Python script that contains one function so that the computational operations will be performed in the Techila environment  Demonstrate the difference between Local Control Code and Worker Code in a Python environment  Demonstrate the basic syntax of the peach-function in a Python environment The material discussed in this example is located in the following folder in the Techila SDK:  techila\examples\python\Tutorial\1_distribution Locally executable Python function The Python script 'local_function.py' contains one function called 'local_function', which consists of one for-loop.

The algorithm of the locally executable function used in this example is shown below.

def local_function(x): result = [] for j in range(x): result.append(1 + 1) return result The function takes one input argument, which defines the number of iterations that will be performed in the for-loop. Every iteration performs the same arithmetic operation: 1+1. The result of the latest iteration will be appended to the 'result' list. For example, when performing five iterations the 'result' list would contain the following values. To execute the locally executable function, use the commands shown below: execfile('local_function.py') result=local_function(5) The output generated by the locally executable program should resemble the one shown below in Figure 3.

Figure 3. Executing the locally executable function in Python. Number of iterations: 5 index 0 1 2 3 4 value 2 2 2 2 2