Source code layout

The source code of GridCompute is split into following modules:

main.py

This module is the start-up script used to launch GridCompute.

It first ensures that there is only one instance running, then creates the interface and initializes all parameters required to run the program. To finish, it runs the main loop associated to interface.

main.ensure_single_instance(current_pid, event_queue)

Ensure that only one instance of the program is running.

Check current pid against a configuration file that will contain pid of other running instance if any.

Parameters:
  • current_pid (int) – pid of current process.
  • event_queue (Queue) – queue of events to process by gui.
main.start(gui, current_pid)

Start the program.

Ensures there is only one instance, initialize all server related parameters and populate interface.

Parameters:
  • gui (GUI) – GUI instance handling program interface.
  • current_pid (int) – pid of main thread.

g_config.py

This module gathers configuration variables used in the other modules.

It is used to share global variables and edit configuration variables easier.

g_config.program_name

Name of the program.

g_config.version

Current version.

g_config.author

Author of the program.

g_config.copyright

Copyright.

g_config.title_windows

Title displayed on windows.

g_config.max_number_process

Maximal number of parallel process that can be executed on this computer.

g_config.gui_refresh_interval

Refresh rate of interface in milliseconds.

g_config.db_connect_frequency

Time in seconds before database is accessed again if no case/result is found.

g_config.db_heartbeat_frequency

Frequency in seconds that heartbeat are sent on running processes to notify database that they are still alive.

g_config.db_heatbeat_dead

Time in seconds without heartbeat after which we consider a process is dead.

g_config.daemon_pause

Time in seconds between each process of daemons.

g_config.log_path

Path of the log file.

g_config.pid_file

Path of the file keeping pid of the program to ensure there is only one single instance running.

g_interface.py

This module contains all GUI functionalities.

It generates the main interface and handles any event that needs to be communicated to the user.

class g_interface.GUI

Class handling GridCompute interface.

It contains every parameter and module required for GUI. At creation, it displays a progress bar.

Parameters:
  • root – main Tk instance
  • application – application selected in gui
  • app_combobox – widget associated with application
  • cases_dict (dict) – dictionary where key is the id of tree item and value is the list of input files associated to the case
  • cases_refresh_label – label identifying time of refresh of “my cases”
  • cases_status_label – label identifying status of refresh of “my cases”
  • dedicated_process – number of dedicated process selected in gui
  • event_queue – queue of events to process by gui
  • log – gui element associated to logging
  • my_process_pid_id (dict) – dictionary of pid to gui id for treeview in “my processes”
  • progress_bar – progress bar in progress window
  • progress_label – text present on progress window
  • progress_window – top level window showing progress of current task set to None if progress window not existing or closed
  • server – Server instance containing main functionalities
  • tree_cases – widget containing cases to submit
  • tree_my_cases – widget containing “my cases”
  • tree_my_process – widget containing “my processes”
add_cases()

Display a window to select cases to submit to server and adds them on the list.

askokcancel(msg)

Ask a question to user.

Parameters:msg (str) – Message to be displayed.
Returns:True if user enters “ok”, False otherwise.
Return type:bool
create_progress_window(progress_mode, progress_text, progress_max=100)

Create a progress window.

The progress window can be controlled afterwards through the event_queue parameter. Refer to function “handle_next_event”.

Parameters:
  • progress_mode (str) – “determinate” if progress level evolution needs to be controlled or “indeterminate”.
  • progress_text (str) – text displayed initially on the progress window.
  • progress_max (int) – argument equal to maximal progress value when complete.
create_report()

Create a full report from database containing all cases from same “user group”. Ask user where he wants to save the report.

error(msg)

Display error on screen.

Parameters:msg (str) – Message to be displayed.
exit_program()

Communicate to processes to exit program.

handle_next_event()

Handles the next event from communicated to the GUI through event_queue variable.

Each element in event_queue is a dictionary. Action performed depends on value of type key:

  • log_file_only: log a message in the log file, no display in gui
  • warning: display a warning
  • info: display an information in gui, optionally creating an info box
  • error: display an error
  • critical: display an error and exit program
  • change progress max: change value of progress window corresponding to completion
  • change progress: modify progress bar level and text
  • close progress: close progress window
  • add case: add a case in “send cases” tab
  • submitted case: show a case as submitted
  • terminate process?: ask user if he wants to terminate all processes
  • send answer through the connection pipe to daemon process
  • add my case: add a case in “my cases” tab
  • add my process: add a process in “my processes” tab
  • remove my process: remove a process from “my processes” tab
  • change my process: change the status of a process in “my processes” tab
  • populate: GUI can be fully populated
  • exit: close main window
Raises:queue.empty: event_queue is empty, there are no more events to process. –
info(msg)

Display info on screen

Parameters:msg (str) – Message to be displayed.

Associate GUI to a Server instance.

Parameters:server – Server instance.
open_about()

Display program information.

open_help()

Display a help message.

open_license()

Display licenses used by the program.

populate()

Populate GUI during loading of program.

refresh()

Refresh GUI at regular intervals.

refresh_my_cases()

Refresh the list of “my cases”.

Access database and display details of user cases that have not been received yet.

remove_selected()

Removes selected cases to submit from the list.

submit_to_server()

Send all cases (not yet submitted) from the list to the server.

warning(msg)

Display warning on screen

Parameters:msg (str) – Message to be displayed.
g_interface.init_log()

Initialize logging in file.

Create necessary directory structure and remove previous log file.

g_interface.write_log(log_message, gui_log=None)

Write log messages in log file and in GUI.

Parameters:
  • log_message – message to be displayed.
  • gui_log – widget receiving log message. None if message not to be displayed in gui.

g_server_management.py

This module contains all server related functionalities.

It creates server variables associated to file server and mongo database and handle the communication with them.

class g_server_management.Server(event_queue, dedicated_process)

Class handling server-related functionalities.

File server path is taken from “server.txt” present at root of program. Settings are taken from “settings.txt” present on the file server.

Server properties are set at initialization.

Parameters:
  • event_queue – queue of events to process by gui
  • gui_dedicated_process – number of dedicated process selected in gui
  • daemon_dedicated_process – number of dedicated process communicated to daemon process
  • exit_program – variable scanned by daemons to know when to exit
  • gui_answer – notify when gui answered a question
  • server_path – path of file server as defined in “server.txt” file
  • app_path – path of applications-specific scripts
  • settings_path – path of settings.txt file
  • settings –

    list of settings from settings.txt on file server including:

    • mongodb server: Address of the mongo instance including connection port containing “gridcompute” like mongodbserver.com:888 or 10.0.0.1:888 or Machine123:888.
    • user group: Login used to connect on mongo database.
    • password: Password used to connect on mongo database.
    • instance: Data instance to consider like 0 or debug.
  • mongodb – Connection to mongo database
  • server_functions –

    Dictionary of application-specific scripts. The key is application name and value is a dictionary of following keys:

    • path: path of application folder
    • send, process, receive: booleans corresponding to existence of these functions
  • software_allowed_to_run – set of applications that can run as defined in “Software_Per_Machine.csv”
access_mongodb()

Access the mongo database.

add_cases(files_selected, application, keep_running=True)

Add a list of cases to interface from files selected by user.

A file selected can correspond to one or several cases to run. Each case being a list of input files. Those cases are sent to the interface which displays the first input file of each case. The application specific “send.py” function is used on each file selected.

Parameters:
  • files_selected – Files selected by user
  • application – Application associated to input files
  • keep_running – Argument that can be associated to the state of a variable. When this variable is False, function ends. It is used to catch when user closes the progress window.
applications_with_process()

Return a dictionary of applications that have a process function.

The key is the application name and the value is the script path.

applications_with_receive()

Return a dictionary of applications that have a receive function.

The key is the application name and the value is the script path.

applications_with_send()

Return a dictionary of applications that have a send function.

The key is the application name and the value is the script path.

create_daemons()

Create daemons required in the application.

Two daemons are created:

  • Daemon process: scans continuously database to check if there are new calculations to perform (if number of processes selected allows it).
  • Daemon receive: scans continuously database to check if there are new results to receive.
create_report(file_report, keep_running=True)

Create a report of cases present on database.

Display cases and their details from database, limited to the ones from same “user group”.

Parameters:keep_running – Argument that can be associated to the state of a variable. When this variable is False, function ends. It is used to catch when user closes the progress window.
exit_processes()

Terminate program.

Ensure daemon process (and its child processes) and daemon receive terminate properly.

get_settings()

Initialize settings variable from “settings.txt” present on server.

get_software_allowed_to_run()

Return the list of software allowed to run on this machine per “Software_Per_Machine.csv” present on file server.

handle_software_permissions()

Verify that mongo database allows current version of program to run.

This is based on the collection “versions” present in mongo database. If “versions” collection is not present, a warning is displayed.

notify_number_process_daemon(*args)

Notify the daemon process of a change of number of processes selected in GUI.

refresh_my_cases(keep_running=True)

Refresh “my cases” list.

Access mongo database to display all user cases that have not been totally processed yet (ie not received by user).

Parameters:keep_running – Argument that can be associated to the state of a variable. When this variable is False, function ends. It is used to catch when user closes the progress window.
scan_applications()

Scan application specific scripts present on file server.

submit_to_server(cases_to_submit, application, keep_running=True)

Submit cases to the server.

Input files are zipped and copied to the file server. Cases are entered in mongo database.

Parameters:
  • cases_to_submit – List of cases. Each case is a list of input files associated to a case.
  • application – Application associated to input files.
  • keep_running – Argument that can be associated to the state of a variable. When this variable is False, function ends. It is used to catch when user closes the progress window.
g_server_management.check_quit_program(exit_program)

Function used by daemons to check if they need to terminate and kill their child processes.

Parameters:exit_program – Variable scanned by daemons to know when to exit.
g_server_management.launch_process(case, event_queue, server_path, settings)

Launch one process from database.

Mongo database is updated when process finishes.

Parameters:
  • case – case to process obtained from mongo database.
  • event_queue – queue of events to process by gui.
  • server_path – path of file server as defined in “server.txt” file.
  • settings – list of settings from settings.txt on file server.
g_server_management.refresh_status_daemon_process(alive_process, dedicated_process, event_queue, exit_program, gui_answer, paused_process)

Refresh status variables used by daemon process.

Parameters:
  • alive_process – List of processes alive.
  • dedicated_process – Number of dedicated process selected in gui.
  • event_queue – Queue of events to process by gui.
  • exit_program – Variable scanned by daemons to know when to exit.
  • gui_answer – Notify when gui answered a question.
  • paused_process – List of processes currently on pause.
g_server_management.return_module(application, function)

import app-specific function send, process or receive

g_server_management.run_daemon_process(applications_with_process, dedicated_process, event_queue, exit_program, gui_answer, server_path, settings, software_allowed_to_run)

Run “daemon process” that launches new processes when possible.

Daemon process runs continuously and check on database if processes are available to launch when user allows it.

Parameters:
  • applications_with_process – dictionary of applications that have a process function
  • dedicated_process – number of dedicated process selected in gui
  • event_queue – queue of events to process by gui
  • exit_program – variable scanned by daemons to know when to exit
  • gui_answer – notify when gui answered a question
  • server_path – path of file server as defined in “server.txt” file
  • settings – list of settings from settings.txt on file server
  • software_allowed_to_run – set of applications that can run as defined in “Software_Per_Machine.csv”
g_server_management.run_daemon_receive(applications_with_receive, event_queue, exit_program, server_path, settings)

Run “daemon receive” that retrieve back cases from server after they have been processed.

Daemon receive runs continuously and check on database if output files are available.

Parameters:
  • applications_with_receive – dictionary of applications that have a process function.
  • event_queue – queue of events to process by gui.
  • exit_program – variable scanned by daemons to know when to exit.
  • server_path – path of file server as defined in “server.txt” file.
  • settings – list of settings from settings.txt on file server.

setup.py

This script is used for building the executuble with cx_Freeze.

Please refer to the section “Development” of the documentation for building the application.

database_management.py

This module contains administrator functions for database management.

database_management.set_up_mongodb_server(mongodb_server, login, password, versions)

Sets up a mongodb server for GridCompute.

Mongo database “gridcompute” is initialized and the “versions” collection is created to specify the program versions that are authorized by the database.

The “gridcompute” database must be present on the server. Any collection in it will be removed.

Parameters:
  • mongodb_server – Address of the mongo instance including connection port containing gridcompute database like mongodbserver.com:888 or 10.0.0.1:888 or Machine123:888
  • login – Login used to connect on mongo database.
  • password – Password used to connect on mongo database.
  • versions –

    List of versions of gridcompute that the mongo database recognizes defined by:

    • _id: version number (ex: ‘0.1’).
    • status: either “allowed”, “warning” or “refused”.
    • message: message to be displayed when status is not “allowed” like:
      [{'_id':'0.1', status:"warning", message:"Beta version},
 {'_id':'1.0', status:"allowed"}]