Difference between revisions of "Dynamo in Amazon EC2"
(14 intermediate revisions by the same user not shown) | |||
Line 10: | Line 10: | ||
[[File:Ec2Services.png |thumb|center|500px|selecting the EC2 services]] | [[File:Ec2Services.png |thumb|center|500px|selecting the EC2 services]] | ||
− | + | The EC2 dashboard is accessed through ''Services > EC2" | |
− | [[File: | + | [[ File:Ec2Dashboard.png|thumb|center|500px| The EC2 dashboard]] |
=== Configuring an instance === | === Configuring an instance === | ||
Line 17: | Line 17: | ||
[[File:Ec2LaunchInstance.png |thumb|center|500px|Launch an instance]] | [[File:Ec2LaunchInstance.png |thumb|center|500px|Launch an instance]] | ||
− | Your instance will be built on the top of | + | |
+ | Your instance will be built on the top of an AMI of type CUDA7.5 provided by NIVIDA. You can find it in the Amazon Marketplace. The use of the AMI itself is free of charge: you will only have to pay to Amazon the time you keep the instance running. | ||
+ | |||
+ | [[ File:Ec2CUDAAMI.png |thumb|center|700px| Select the CUDA AMI from the marketplace]] | ||
+ | |||
+ | After the selection, some additional information from Nvidia will pop up: | ||
+ | |||
+ | [[ File:Ec2NvidiaSelected.png |thumb|center|700px|Launch an instance]] | ||
+ | |||
+ | ==== Instance type ==== | ||
+ | |||
+ | Make certain that you initiate an instance type that includes a GPU machine. Amzon describes [[https://aws.amazon.com/ec2/instance-types/ here]] its different instance types... please make certain that you check the prizing scheme before launching the instance. | ||
+ | |||
+ | [[ File:Ec2ChooseGPUInstance.png |thumb|center|600px| Select an instance type that includes GPU accelerators]] | ||
+ | |||
+ | For some of the instance types you might need to contact Amazon: the default profile user usually cannot ask for a <tt>p2.large</tt> or an <tt>g2.large</tt> instance. | ||
+ | |||
+ | [[ File:Ec2FailedLaunch.png |thumb|center|600px| Error message when GPU instances are not available for your profile in your zone]] | ||
+ | |||
+ | You'll probably need to apply for permission for launching them using your user support menu. This might take one or two days. | ||
+ | |||
+ | Also, take into account that not all areas provide all types of instances. You might need to change your geographic area in order to access GPU instances. | ||
+ | |||
+ | You will get an (alarming looking) warning. This is simply telling you that your GPU instance is not free of charge and that you will incur costs if you proceed with the launch of the instance. So, be aware of this and discard/ignore the message. | ||
+ | [[ File:Ec2NonFreeTier.png |thumb|center|600px| Warning about non elegibility for free tier user can be ignored.]] | ||
+ | |||
+ | ==== Storage ==== | ||
+ | |||
+ | Make certain that your instance has at least 16Gb of storage capacity. The default of 8G is mainly consumed by the OS, and in the remaining size you will not be able to download and untar the ''Dynamo'' package. | ||
+ | |||
+ | [[ File:AddStorageMain.png |thumb|center|500px|Selecting adequate storage]] | ||
+ | |||
+ | |||
+ | ==== Connection keys ==== | ||
+ | |||
+ | When you press on ''Launch'', you'll be asked to create a ''key'' file, or to use a previous one (from a previous Amazon EC2 session). This is merely a file <tt>.pem</tt> that you'' need to use in order to connect to your instance, or to transfer data to/from it. | ||
+ | |||
+ | [[ File:Ec2CreateKeys.png |thumb|center|700px|Generating or selecting a key file]] | ||
=== Launching an instance === | === Launching an instance === | ||
− | [[File:Ec2LaunchInstanceInformation.png |thumb|center| | + | After launching the instance, a window will pop up with information for connecting to the virtual server through an <tt>ssh</tt> command. You will get the public DNS of the instance, and some instructions. |
+ | |||
+ | [[File:Ec2LaunchInstanceInformation.png |thumb|center|700px|Information after launching of the instance]] | ||
+ | |||
+ | However, the information is not totally correct, as the user name you need to use is <tt>ec2-user</tt>. This information can be consulted [[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html?icmpid=docs_ec2_console in the documentation provided by Amazon]] | ||
+ | |||
+ | A successful launch will open this site: | ||
− | + | [[File:Ec2SuccesfulLaunch.png |thumb|center|700px|Successful launch]] | |
+ | |||
+ | which will not update. You probably want to go to the dashboard of resources and then click on ''Running Instances'' | ||
+ | |||
+ | [[File:Ec2runningInstancesDashboard.png |thumb|center|700px|Running instances in dashboard ]] | ||
+ | |||
+ | to control the current status of the instance you just launched | ||
+ | |||
+ | [[ File:Ec2InstanceManager.png|thumb|center|700px|Management of instances]] | ||
=== Connecting to the instance === | === Connecting to the instance === | ||
Line 36: | Line 87: | ||
Here, the instance location should look something like <tt>ec2-52-214-203-126.eu-west-1.compute.amazonaws.com</tt> | Here, the instance location should look something like <tt>ec2-52-214-203-126.eu-west-1.compute.amazonaws.com</tt> | ||
+ | |||
+ | After you connect to it, you can check that the nvidia driver is in place by typing | ||
+ | |||
+ | <tt>nvidia-smi</tt> | ||
+ | |||
+ | You can also find the CUDA libraries under <tt>/usr/local/cuda</tt> | ||
== Running Dynamo projects== | == Running Dynamo projects== | ||
− | |||
− | |||
=== Copying Dynamo === | === Copying Dynamo === | ||
− | + | First, you need to transfer the ''Dynamo'' package. Assuming you have the <tt>tar</tt> file in your local machine: | |
− | <tt> scp -i "keyFile.pem" <your Dynamo tar> <your | + | <tt> scp -i "keyFile.pem" <your Dynamo tar> <ec2-user@<your instance's public DNS>:/home/ubuntu/ </tt> |
=== Setting the terminal === | === Setting the terminal === | ||
− | Then, you can ssh to your EC2 terminal. There, you still | + | Then, you can <tt>ssh</tt> to your EC2 virtual terminal. There, you still to install ''Dynamo'': |
<tt>mkdir dynamo</tt> | <tt>mkdir dynamo</tt> | ||
Line 59: | Line 114: | ||
<tt>sudo yum install -y libXt </tt> | <tt>sudo yum install -y libXt </tt> | ||
− | Now you are ready to go. This terminal will be able to run ''Dynamo'' projects in the usual way. | + | Now you are ready to go. This terminal will be able to run ''Dynamo'' projects in the usual way. We suggest you to test the installation by activating ''Dynamo'' |
+ | |||
+ | <tt>source ~/dynamo/activateLinux.sh</tt> | ||
+ | |||
+ | and the opening its command line: | ||
+ | |||
+ | <tt>dynamo</tt> | ||
+ | |||
+ | and then creating a synthetic project | ||
+ | |||
+ | <tt>Dynamo > dtutorial mytest -p pgpu </tt> | ||
+ | |||
+ | Then we adapt it for running in the GPU: | ||
+ | |||
+ | <tt>Dynamo > dvput pgpu dst standalone_gpu </tt> | ||
+ | |||
+ | We can also select the GPU identifiers: | ||
+ | |||
+ | <tt>Dynamo > dvput pgpu -gpus [0,1] </tt> | ||
+ | |||
+ | After that, we need to unfold the project to make the changes permanent | ||
+ | |||
+ | <tt>Dynamo > dvunfold </tt> | ||
+ | |||
+ | execute from the command line of the terminal (outside of the ''Dynamo'' prompt <tt>Dynamo > </tt> ) | ||
+ | |||
+ | <tt>./pgpu.exe</tt> | ||
+ | |||
+ | ==== Graphic output==== | ||
+ | Note that further settings are needed: : | ||
− | |||
<pre>sudo yum install -y libGLU | <pre>sudo yum install -y libGLU | ||
sudo yum install -y libXrand</pre> | sudo yum install -y libXrand</pre> | ||
+ | |||
+ | This will merely prepare your system to talk to the graphical libraries delivered by ''Dynamo''. Still you need to prepare a X forwarding protocol between your local machine and your Amazon terminal | ||
+ | |||
+ | == Killing the instance == | ||
+ | |||
+ | A running instance is consuming money. If you stop it, you will still pay for the storage of the data in the terminal. In order to totally kill the instance, including stored data, you need to ''terminate'' it. | ||
+ | |||
+ | [[ File:Ec2Terminate.png |thumb|center|800px|Terminate an instance]] |
Latest revision as of 08:56, 23 October 2017
We are preparing a release of Dynamo for the EC2 cloud computing service. This will allow users without access to a graphic accelerator to use Dynamo on a powerful GPU in the cloud, paying only for the actually used computing time, and avoid the purchase, installation and maintenance of special hardware.
The Dynamo AMI itself can be used free of charge and needs no additional license. It delivers a tested installation of all needed software (Dynamo, CUDA, MCR libraries), ready for immediate use upon connection to your account.
Feel free to contact us if you want to be a beta tester.
Contents
Creating an account
We will post here a walkthrough for the creation of an user account in the Amazon EC2.
The EC2 dashboard is accessed through Services > EC2"
Configuring an instance
Your instance will be built on the top of an AMI of type CUDA7.5 provided by NIVIDA. You can find it in the Amazon Marketplace. The use of the AMI itself is free of charge: you will only have to pay to Amazon the time you keep the instance running.
After the selection, some additional information from Nvidia will pop up:
Instance type
Make certain that you initiate an instance type that includes a GPU machine. Amzon describes [here] its different instance types... please make certain that you check the prizing scheme before launching the instance.
For some of the instance types you might need to contact Amazon: the default profile user usually cannot ask for a p2.large or an g2.large instance.
You'll probably need to apply for permission for launching them using your user support menu. This might take one or two days.
Also, take into account that not all areas provide all types of instances. You might need to change your geographic area in order to access GPU instances.
You will get an (alarming looking) warning. This is simply telling you that your GPU instance is not free of charge and that you will incur costs if you proceed with the launch of the instance. So, be aware of this and discard/ignore the message.
Storage
Make certain that your instance has at least 16Gb of storage capacity. The default of 8G is mainly consumed by the OS, and in the remaining size you will not be able to download and untar the Dynamo package.
Connection keys
When you press on Launch, you'll be asked to create a key file, or to use a previous one (from a previous Amazon EC2 session). This is merely a file .pem that you need to use in order to connect to your instance, or to transfer data to/from it.
Launching an instance
After launching the instance, a window will pop up with information for connecting to the virtual server through an ssh command. You will get the public DNS of the instance, and some instructions.
However, the information is not totally correct, as the user name you need to use is ec2-user. This information can be consulted [in the documentation provided by Amazon]
A successful launch will open this site:
which will not update. You probably want to go to the dashboard of resources and then click on Running Instances
to control the current status of the instance you just launched
Connecting to the instance
Once your instance is running, you can connect to it through an ssh terminal. Typically, you need to include the key file generated in the moment of launching the instance. Remember that you 'need to ensurer that your key is not publicly viewable through chmod
chmod 400 myKey.pem
Then you can connect to your instance of the NVIDIA AMI
ssh -i "myKey.pem" ec2-user@<instance public DNS>
Here, the instance location should look something like ec2-52-214-203-126.eu-west-1.compute.amazonaws.com
After you connect to it, you can check that the nvidia driver is in place by typing
nvidia-smi
You can also find the CUDA libraries under /usr/local/cuda
Running Dynamo projects
Copying Dynamo
First, you need to transfer the Dynamo package. Assuming you have the tar file in your local machine:
scp -i "keyFile.pem" <your Dynamo tar> <ec2-user@<your instance's public DNS>:/home/ubuntu/
Setting the terminal
Then, you can ssh to your EC2 virtual terminal. There, you still to install Dynamo:
mkdir dynamo
tar -xf <Dynamo tar file> -C /home/ubuntu/dynamo
You need to make an extra couple of adjustments on your terminal. The AMI provided by envidia needs to be able to talk to the Matlab libraries provided by your Dynamo tar.
sudo yum install -y libXt
Now you are ready to go. This terminal will be able to run Dynamo projects in the usual way. We suggest you to test the installation by activating Dynamo
source ~/dynamo/activateLinux.sh
and the opening its command line:
dynamo
and then creating a synthetic project
Dynamo > dtutorial mytest -p pgpu
Then we adapt it for running in the GPU:
Dynamo > dvput pgpu dst standalone_gpu
We can also select the GPU identifiers:
Dynamo > dvput pgpu -gpus [0,1]
After that, we need to unfold the project to make the changes permanent
Dynamo > dvunfold
execute from the command line of the terminal (outside of the Dynamo prompt Dynamo > )
./pgpu.exe
Graphic output
Note that further settings are needed: :
sudo yum install -y libGLU sudo yum install -y libXrand
This will merely prepare your system to talk to the graphical libraries delivered by Dynamo. Still you need to prepare a X forwarding protocol between your local machine and your Amazon terminal
Killing the instance
A running instance is consuming money. If you stop it, you will still pay for the storage of the data in the terminal. In order to totally kill the instance, including stored data, you need to terminate it.