iMath Cloud User's Manual

(current version: dawn-beta-1.0.0)

Introduction

Thanks for your interest in our service. iMath Cloud is a web-based system for data scientists and data analysts that uses Python-related technologies. The current version is in beta and completely free of charge. Feel free to try it out and enjoy the set of free computational resources deployed in the cloud for you. In the current version you can

  • Create and share projects: Use our iMath Connect tool to create new projects, assign computational resources and share them with your colleagues or collaborators.
  • Edit and run code files in several interactive consoles: Use iMath Cloud to upload, create and edit files in our highlighted editor right in your navigator. Execute the code to our interactive console, a seamless integration of IPython Notebook.
  • Start as many consoles as you want: Use iMath Cloud to start new consoles on demand and work in parallel. The code is executed in our servers using the computational resources of your choice. New in this version: From this version we enable in-line visualizations of animations created with FuncAnimation of matplotlib and streaming in-line visualizations of mp4 files.
  • Submit intensive Jobs: Use iMath Cloud to submit your files as jobs in the background. They will be executed in our servers using the computational resources of your choice. All generated files will remain in the cloud and two more extra files will be generated: One for the standard output and another for the standard error output.

Getting started

The application is accessible through this link. Log-in or register as a new user in 2 seconds and access iMath Connect (see below) in order to manage and create new projects, and invite collaborators.

You only need a web navigator and a desktop computer, laptop, tablet or any mobile device you want. The front-end has been optimized for Mozilla Firefox (version 30 and above) and Google Chrome (version 37.0.2062.120 and above).

Managing Projects: iMath Connect

Main Concepts

iMath Connect is the web-based system devoted to the management of projects, computational resources and collaborators. From iMath Connect you can:

  • Create/Update/Delete projects.
  • Assign projects to computational resources.
  • Add/Remove collaborators to your projects.
  • See the projects you are collaborating with not as owner.
  • Start iMath Cloud, the working environment.

Some concepts:

  • Instance: A group of computational resources: Number of CPUs, RAM, Storage
    • An instance can be public or private. Public instances are offered for free by altruist institutions. Private Instances are managed by the user. NOTE:In the current version, no private instances can be created.
    • An instance can host several projects.
    • If only one project is active in a given instance, the project will use all available computational resources (CPUs and RAM). Storage will be shared by the other projects assigned to the same instance.

* Project: An entity that compiles a set of data files and the files that are deployed in the cloud.

  • Each project is independent from each other.
  • Each project has an owner user.
  • A project can have collaborators. The owner of the project is the only one that can add/remove collaborators to the project.
  • A project can be removed. The owner of the project is the only one that can do this.
  • Each project is assigned to one and only one instance.
  • The working environment of a given project is managed by iMath Cloud, an independent platform from iMath Connect.

General and Dashboard View

Dashboard view of iMath Connect

hte header of the general view provides direct access to (1) this wiki, (2) the notification system and (3) the profile which allows to update user's picture and change password.

Access to the notification system and the profile options.

The dashboard also offers a general view of the projects, collaborations and instances.

My Projects

The projects managed by the user. From the table the user can easily access iMath Cloud by clicking the triangle, access the detail of the project by clicking in the link, and remove the project from the system by clicking the remove button.

My Projects table

Collaborations

The projects that the user collaborates with. From here the user can as easily access iMath Cloud by clicking the triangle.

Collaborations table

My Instances

The instances managed by the user (not enabled in this version)

Public Instances

The instances that can be used by the user and are provided by an altruist institution. All resources are specified and no actions can be taken from here in this version.

Public instance table

My Projects View

My Projects view of iMath Conect

This view offers CRUD operations for projects, including the capability to add collaborators.

My Projects

In this view, the table of projects appears as well, with the exact same possible actions as in the dashboard. The button New enables the creation of a project (see below).

My Projects table

Adding collaborators and Updating Description

On project selection there is the possibility to add collaborators and update the description of the project. In this version neither the name of the project nor the assigned instance can be changed. Button Save performs such action.

Button Add Collaborator adds a new collaborator. One can type a current user name or an email. If the user name or the email correspond to an existing user, it becomes a new collaborator of the project. If an email is provided and it is not found, an email is sent to the recipient announcing the invitation and the credentials to access to the system.

Update project data

Creating projects

When button New is clicked it is possible to create a project.

Creating a new project

A project name must be given (only alphanumerical characters), a description and an instance must be selected. Pressing Select Resources the following screen appears, which enables the selection of computational resources:

Selecting an instance

Working with Projects: iMath Cloud

The imath Cloud interface

Main Concepts

iMath Cloud is the web-based developing environment to work within a project. It provides basic and standard methods for managing and updating files and edit them, and also highlights Python, R or Octave code when a .py,.r and .m file is being edited. Moreover, the system offers a seamless integration between IPython Notebook and the file management system, and offers a job management system as well. Here are some concepts:

  • Files and Directories: Each project has a cloud storage to host files and directories. Through the file tree view (see below) users have access to CRUD operations. Among files, there are some special ones:
    • .py: Python files. When edited, they are highlighted. Also, they can be submitted as a Job (see below) or send to the interactive math console.
    • .r: R files. When edited, they are highlighted, although it is not possible to execute them in the current version.
    • .m: Octave files. When edited, they are highlighted, although it is not possible to execute them in the current version.
    • .csv: Comma-separated value files. Can be plotted directly from the tree view and enable a set of simple statistical information about the data. (Note: In the current version only numerical values can be considered for the build-in operations).
    • .ipynb: IPython Notebook files. Since several consoles can be opened, the files of the notebook can be stored as well in the cloud. There is always a default notebook by default, although the other ones can be opened by clicking the IPython notebook file.
  • Jobs: They are initiated by a single Python file or by a build-in operation over a .csv file. They are executed in background and are thought for intensive computations. All generated files as well as the standard output and standard error output are captured by the system and placed in the cloud storage of the project. Once a job is submitted, it has a state. Here is the state diagram of a job:

Job states and transitions

  • Interactive Math Console: A shell script to execute Python code. We have integrated IPython Notebook. More than one console can be opened and used in parallel.

File Management System

File tree

It offers the standard CRUD operations over files and directories plus other functionalities.

  • On Directories (right click)
    • Rename: Rename the directory.
    • Delete: Delete the directory.
    • Copy: Copy the directory.
    • Paste: Paste a previously copied directory or file.
    • New Directory: Creates a new directory inside the selected directory.
    • New File: Creates an empty file in the selected directory.
    • Upload Files: Upload files to the cloud from local to the selected directory
    • Download as zip: Download the directory as a zip file.
  • On Files (left or right click)
    • Edit: Opens the file and enables edition.
    • Rename: Rename the file.
    • Delete: Delete the file.
    • Copy: Copy the file.
    • Block: Block the file.
    • Unblock: Unblock the file.
    • Download as zip: Download the file as a zip file .zip.
    • Run on Console: Runs the code of the file in the current active console (ONLY PYTHON FILES .py)
    • Run as job: Runs the Python file as a job (ONLY PYTHON FILES .py)
    • Open Console: Opens the corresponding IPython Notebook console (ONLY IPYTHON NOTEBOOK FILES .ipynb)
    • Math: Execute as job the corresponding operation (ONLY CSV FILES .csv)
  • Buttons (from left to right)
    • Refresh: Refresh the tree view. It shows an exact structure of the cloud files.

Block and unblock files

Blocking and unblocking files with the aim of controlling when several users are sharing the same project. It should be noted several aspects related with blocking and unblocking consoles and files:

  • When a console is opened, its associated file is automatically blocked, and in the same way, when the console is closed, its associated file is unblocked.
  • When a console is opened, the user cannot unblock its associated file.
  • The default console cannot be blocked.
  • When a file is blocked by an user, the rest of the users, sharing this file, only can display it, cannot edit it.
  • An icon appears close to the name of the file (in the tree file) that indicates if the file is blocked. Hover the pointer over the icon, you get information about who have blocked the file. If the color of the icon is green, it indicates the current user has blocked the file; however, if the color is red, it indicates that the file has been blocked by other user different to the current one.

Editing files

File editing

Allows editing any text file. in particular, it highlights Python, R or Octave code when a .py, .r and .m files are opened respectively.

  • Buttons (from left to right)
    • Save: Save the content of the file.
    • Submit to Console: Submit the content of the file to the active math console. Only enabled for Python files in the current version.
    • Submit as Job: Submit the file as a background job. Enabled for Python files in the current version.

IPython Notebook Integration

 IPython Notebook integration

There is a seamless integration with IPython Notebook (version 0.13.1). We have modified the necessary elements to integrate it in iMath Cloud and to ensure that it is completely secured. All documentation can be found in the official IPython Notebook site.

The interactive console permits to open as many notebook as you want. When the user opens a new one, it has to select which kind of mathematical language it needs. Now, it is possible to select from python, R or Octave.

R new console

Octave new console

Store images

When any user has created a new image, it can store in the File System automatically dropping the image from the interactive console to File System. The associated name to each figure will be Image_X.png. The X's value will be incrementing each time.

Animations

NEW

In this version we allow in-line visualization of animations in the same notebook created with FuncAnimation of Matplotlib. We have integrated External Link|JS Animator and add new features to enable in-line visualization of animations. Here is how it looks like:

Animation with FuncAnimation

If the generated animation is too big to be displayed in an embedded responsive web application (2 MB aprox.) we visualize part of the animation and suggest further actions to visualize it entirely.

Animation with FuncAnimation too big

When saving the video in mp4 we use html5 features to visualize it in streaming.

MP4 video generation and animation

Here is the example we use

anim_example.py
"""
Animation of Elastic collisions with Gravity
 
author: Jake Vanderplas
email: vanderplas@astro.washington.edu
website: http://jakevdp.github.com
license: BSD
Please feel free to use and modify this, but keep the above information. Thanks!
"""
import numpy as np
from scipy.spatial.distance import pdist, squareform
 
import matplotlib.pyplot as plt
import scipy.integrate as integrate
import matplotlib.animation as animation
 
class ParticleBox:
    """Orbits class
 
    init_state is an [N x 4] array, where N is the number of particles:
       [[x1, y1, vx1, vy1],
        [x2, y2, vx2, vy2],
        ...               ]
 
    bounds is the size of the box: [xmin, xmax, ymin, ymax]
    """
    def __init__(self,
                 init_state = [[1, 0, 0, -1],
                               [-0.5, 0.5, 0.5, 0.5],
                               [-0.5, -0.5, -0.5, 0.5]],
                 bounds = [-2, 2, -2, 2],
                 size = 0.04,
                 M = 0.05,
                 G = 9.8):
        self.init_state = np.asarray(init_state, dtype=float)
        self.M = M * np.ones(self.init_state.shape[0])
        self.size = size
        self.state = self.init_state.copy()
        self.time_elapsed = 0
        self.bounds = bounds
        self.G = G
 
    def step(self, dt):
        """step once by dt seconds"""
        self.time_elapsed += dt
 
        # update positions
        self.state[:, :2] += dt * self.state[:, 2:]
 
        # find pairs of particles undergoing a collision
        D = squareform(pdist(self.state[:, :2]))
        ind1, ind2 = np.where(D < 2 * self.size)
        unique = (ind1 < ind2)
        ind1 = ind1[unique]
        ind2 = ind2[unique]
 
        # update velocities of colliding pairs
        for i1, i2 in zip(ind1, ind2):
            # mass
            m1 = self.M[i1]
            m2 = self.M[i2]
 
            # location vector
            r1 = self.state[i1, :2]
            r2 = self.state[i2, :2]
 
            # velocity vector
            v1 = self.state[i1, 2:]
            v2 = self.state[i2, 2:]
 
            # relative location & velocity vectors
            r_rel = r1 - r2
            v_rel = v1 - v2
 
            # momentum vector of the center of mass
            v_cm = (m1 * v1 + m2 * v2) / (m1 + m2)
 
            # collisions of spheres reflect v_rel over r_rel
            rr_rel = np.dot(r_rel, r_rel)
            vr_rel = np.dot(v_rel, r_rel)
            v_rel = 2 * r_rel * vr_rel / rr_rel - v_rel
 
            # assign new velocities
            self.state[i1, 2:] = v_cm + v_rel * m2 / (m1 + m2)
            self.state[i2, 2:] = v_cm - v_rel * m1 / (m1 + m2) 
 
        # check for crossing boundary
        crossed_x1 = (self.state[:, 0] < self.bounds[0] + self.size)
        crossed_x2 = (self.state[:, 0] > self.bounds[1] - self.size)
        crossed_y1 = (self.state[:, 1] < self.bounds[2] + self.size)
        crossed_y2 = (self.state[:, 1] > self.bounds[3] - self.size)
 
        self.state[crossed_x1, 0] = self.bounds[0] + self.size
        self.state[crossed_x2, 0] = self.bounds[1] - self.size
 
        self.state[crossed_y1, 1] = self.bounds[2] + self.size
        self.state[crossed_y2, 1] = self.bounds[3] - self.size
 
        self.state[crossed_x1 | crossed_x2, 2] *= -1
        self.state[crossed_y1 | crossed_y2, 3] *= -1
 
        # add gravity
        self.state[:, 3] -= self.M * self.G * dt
 
 
#------------------------------------------------------------
# set up initial state
np.random.seed(0)
init_state = -0.5 + np.random.random((50, 4))
init_state[:, :2] *= 3.9
 
box = ParticleBox(init_state, size=0.04)
dt = 1. / 30 # 30fps
 
 
#------------------------------------------------------------
# set up figure and animation
fig = plt.figure()
fig.subplots_adjust(left=0, right=1, bottom=0, top=1)
ax = fig.add_subplot(111, aspect='equal', autoscale_on=False,
                     xlim=(-3.2, 3.2), ylim=(-2.4, 2.4))
 
# particles holds the locations of the particles
particles, = ax.plot([], [], 'bo', ms=6)
 
# rect is the box edge
rect = plt.Rectangle(box.bounds[::2],
                     box.bounds[1] - box.bounds[0],
                     box.bounds[3] - box.bounds[2],
                     ec='none', lw=2, fc='none')
ax.add_patch(rect)
 
def init():
    """initialize animation"""
    global box, rect
    particles.set_data([], [])
    rect.set_edgecolor('none')
    return particles, rect
 
def animate(i):
    """perform animation step"""
    global box, rect, dt, ax, fig
    box.step(dt)
 
    ms = int(fig.dpi * 2 * box.size * fig.get_figwidth()
             / np.diff(ax.get_xbound())[0])
 
    # update pieces of the animation
    rect.set_edgecolor('k')
    particles.set_data(box.state[:, 0], box.state[:, 1])
    particles.set_markersize(ms)
    return particles, rect
 
ani = animation.FuncAnimation(fig, animate, frames=500,
                              interval=10, blit=True, init_func=init)
ani.save('my_animation.mp4', extra_args=['-vcodec', 'libx264'])
iMath.display_video('my_animation.mp4')

Job Management System

Jobs List

When a Job is submitted, it will appear in the list of jobs. Such list can be filtered by state and date of submission. Right click on a job row enables the following options:

  • Terminate: Stops the job. The state must be Running.
  • Delete: Delete the job and the associated output files from the system. In the same way, an file that has been generated by a Job cannot be removed. If you want to keep the results of a job, but want to remove the job itself, just copy the output files.
 
public/imathcloud.txt · Last modified: 2015/01/14 14:22 by izubizarreta
 
Except where otherwise noted, content on this wiki is licensed under the following license:CC Attribution-Noncommercial-Share Alike 3.0 Unported
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki