Question? Leave a message!




Parallel and Distributed Computing

Parallel and Distributed Computing
Dr.JakeFinlay Profile Pic
Dr.JakeFinlay,Germany,Teacher
Published Date:22-07-2017
Website URL
Comment
Parallel and Distributed Ccomputing with Julia Marc Moreno Maza University of Western Ontario, London, Ontario (Canada) CS3101 (updated January 7, 2015)Plan 1 Tasks: Concurrent Function Calls 2 Julia's Prnciples for Parallel Computing 3 Tips on Moving Code and Data 4 Around the Parallel Julia Code for Fibonacci 5 Parallel Maps and Reductions 6 Distributed Computing with Arrays: First Examples 7 Distributed Arrays 8 Map Reduce 9 Shared Arrays 10 Matrix Multiplication Using Shared Arrays 11 Synchronization 12 A Simple Simulation Using Distributed ArraysTasks: Concurrent Function Calls Plan 1 Tasks: Concurrent Function Calls 2 Julia's Prnciples for Parallel Computing 3 Tips on Moving Code and Data 4 Around the Parallel Julia Code for Fibonacci 5 Parallel Maps and Reductions 6 Distributed Computing with Arrays: First Examples 7 Distributed Arrays 8 Map Reduce 9 Shared Arrays 10 Matrix Multiplication Using Shared Arrays 11 Synchronization 12 A Simple Simulation Using Distributed ArraysTasks: Concurrent Function Calls Tasks (aka Coroutines) Tasks Tasks are a control ow feature that allows computations to be suspended and resumed in a exible manner This feature is sometimes called by other names, such as symmetric coroutines, lightweight threads, cooperative multitasking, or one-shot continuations. When a piece of computing work (in practice, executing a particular function) is designated as a Task, it becomes possible to interrupt it by switching to another Task. The original Task can later be resumed, at which point it will pick up right where it left o Tasks: Concurrent Function Calls Producer-consumer scheme The producer-consumer scheme One complex procedure is generating values and another complex procedure is consuming them. The consumer cannot simply call a producer function to get a value, because the producer may have more values to generate and so might not yet be ready to return. With tasks, the producer and consumer can both run as long as they need to, passing values back and forth as necessary. Julia provides the functions produce and consume for implementing this scheme.Tasks: Concurrent Function Calls Producer-consumer scheme example function producer() produce("start") for n=1:2 produce(2n) end produce("stop") end To consume values, rst the producer is wrapped in a Task, then consume is called repeatedly on that object: ulia p = Task(producer) Task julia consume(p) "start" julia consume(p) 2 julia consume(p) 4 julia consume(p) "stop"Tasks: Concurrent Function Calls Tasks as iterators A Task can be used as an iterable object in a for loop, in which case the loop variable takes on all the produced values: julia for x in Task(producer) println(x) end start 2 4 stopTasks: Concurrent Function Calls More about tasks julia for x in 1,2,4 println(x) end 1 2 4 julia t = task for x in 1,2,4 println(x) end Task (runnable) 0x00000000045c62e0 julia istaskdone(t) false julia current_task() Task (waiting) 0x00000000041473b0 julia consume(t) 1 2 4 1-element ArrayAny,1: nothingJulia's Prnciples for Parallel Computing Plan 1 Tasks: Concurrent Function Calls 2 Julia's Prnciples for Parallel Computing 3 Tips on Moving Code and Data 4 Around the Parallel Julia Code for Fibonacci 5 Parallel Maps and Reductions 6 Distributed Computing with Arrays: First Examples 7 Distributed Arrays 8 Map Reduce 9 Shared Arrays 10 Matrix Multiplication Using Shared Arrays 11 Synchronization 12 A Simple Simulation Using Distributed ArraysJulia's Prnciples for Parallel Computing Julia's message passing principle Julia's message passing Julia provides a multiprocessing environment based on message passing to allow programs to run on multiple processors in shared or distributed memory. Julias implementation of message passing is one-sided:  the programmer needs to explicitly manage only one processor in a two-processor operation  these operations typically do not look like message send and message receive but rather resemble higher-level operations like calls to user functions.Julia's Prnciples for Parallel Computing Remote references and remote calls Two key notions: remote references and remote calls A remote reference is an object that can be used from any processor to refer to an object stored on a particular processor. A remote call is a request by one processor to call a certain function on certain arguments on another (possibly the same) processor. A remote call returns a remote reference. How remote calls are handled in the program ow Remote calls return immediately: the processor that made the call can then proceeds to its next operation while the remote call happens somewhere else. You can wait for a remote call to nish by calling wait on its remote reference, and you can obtain the full value of the result using fetch.Julia's Prnciples for Parallel Computing Remote references and remote calls: example morenogorgosaurus: julia -p 4 julia r = remotecall(2, rand, 2, 2) RemoteRef(2,1,6) julia fetch(r) 2x2 ArrayFloat64,2: 0.675311 0.735236 0.682474 0.569424 julia s = spawnat 2 1+fetch(r) RemoteRef(2,1,8) julia fetch(s) 2x2 ArrayFloat64,2: 1.67531 1.73524 1.68247 1.56942 Commnets on the example Starting with julia -p n provides n processors on the local machine. The rst argument to remote call is the index of the processor that will do the work. The rst line we asked processor 2 to construct a 2-by-2 random matrix, and in the third line we asked it to add 1 to it. The spawnat macro evaluates the expression in the second argument on the processor speci ed by the rst argument.Julia's Prnciples for Parallel Computing More on remote references julia remotecall_fetch(2, getindex, r, 1, 1) 0.675311345332873 remote call fetch Occasionally you might want a remotely-computed value immediately. The function remotecall fetch exists for this purpose. It is equivalent to fetch(remotecall(...)) but is more ecient. Note that getindex(r,1,1) is equivalent to r1,1, so this call fetches the rst element of the remote reference r.Julia's Prnciples for Parallel Computing The macro spawn The macro spawn The syntax of remote call is not especially convenient. The macro spawn makes things easier:  It operates on an expression rather than a function, and  chooses the processor where to do the operation for you julia r = spawn rand(2,2) RemoteRef(3,1,12) julia s = spawn 1+fetch(r) RemoteRef(3,1,13) julia fetch(s) 2x2 ArrayFloat64,2: 1.6117 1.20542 1.12406 1.51088 Remarks on the example Note that we used 1+fetch(r) instead of 1+r. This is because we do not know where the code will run, so in general a fetch might be required to move r to the processor doing the addition. In this case, spawn is smart enough to perform the computation on the processor that owns r, so the fetch will be a no-op.Tips on Moving Code and Data Plan 1 Tasks: Concurrent Function Calls 2 Julia's Prnciples for Parallel Computing 3 Tips on Moving Code and Data 4 Around the Parallel Julia Code for Fibonacci 5 Parallel Maps and Reductions 6 Distributed Computing with Arrays: First Examples 7 Distributed Arrays 8 Map Reduce 9 Shared Arrays 10 Matrix Multiplication Using Shared Arrays 11 Synchronization 12 A Simple Simulation Using Distributed ArraysTips on Moving Code and Data Availability of a function to processors (1/3) One important point is that your code must be available on any processor that runs it. For example, type the following into the julia prompt julia function rand2(dims...) return 2rand(dims...) end julia rand2(2,2) 2x2 Float64 Array: 0.153756 0.368514 1.15119 0.918912 julia spawn rand2(2,2) RemoteRef(1,1,1) julia spawn rand2(2,2) RemoteRef(2,1,2) julia exception on 2: in anonymous: rand2 not definedTips on Moving Code and Data Availability of a function to processors (2/3) In the previous example, Processor 1 knew about the function rand2, but processor 2 did not. To make your code available to all processors, the require function will automatically load a source le on all currently available processors: julia require("myfile") In a cluster, the contents of the le (and any les loaded recursively) will be sent over the network.Tips on Moving Code and Data Availability of a function to processors (3/3) julia everywhere id = myid() julia remotecall_fetch(2, ()-id) 2 julia workers() 4-element ArrayInt64,1: 2 3 4 5 The everywhere macro executes a statement on all processes.Tips on Moving Code and Data Running Julia with several proocesses or several machines Each process has an associated identi er. The process providing the interactive julia prompt always has an id equal to 1, as would the julia process running the driver script in the example above. The processes used by default for parallel operations are referred to as workers. When there is only one process, process 1 is considered a worker. Otherwise, workers are considered to be all processes other than process 1.Tips on Moving Code and Data Running Julia with several proocesses or several machines The base Julia installation has in-built support for two types of clusters:  A local cluster speci ed with the -p option as shown above.  A cluster spanning machines using the machine le option. This uses a passwordless ssh login to start julia worker processes (from the same path as the current host) on the speci ed machines. Functions addprocs, rmprocs, workers, and others are available as a programmatic means of adding, removing and querying the processes in a cluster.