Google Summer of Code 2017: The Finale!

And, after a beautiful and productive summer, my watch has ended. 
My Google Summer of Code 2017 project was 'Biometric Signal Sensor's Interface' for The Italian Mar's Society, sub-org under Python Software Foundation.

This is a brief summary of my project.

This summer has been wonderful working on the above project. It wouldn't have been wonderful if it wasn't for my GSoC mentors - Antonio Del Mastro, Mario Tambos and Ambar Mehrotra - who helped all throughout the summer, and my GSoC mate and friend Dipankar Niranjan who was selected on the same project, but for a different module that is coupled with mine.

To put forth my experience and work done during my time as a GSoCer, I shall explain about all three phases of the time-line.

The first phase of the project involved researching on various devices and sensor's that existed in the market and to pick up the best and feasible ones for the project. Me and Dipankar started the research together right from the Community Bonding period as we would have plausible academic commitments during the summer.  My research on the sensors and devices almost always ended up in the domain of IoT. But fortunately, Dipankar struck gold and found out about Hexoskin. It was beautifully crafted vest (yes, we did find some issues with it) with good technical support and APIs that was very responsive. This was put in our report of devices and decided to go ahead and place an order for it, after the approval of the mentors. (They loved it!). I also managed to get the code to access the APIs working - their Python client came in handy. The entire architecture was decided and details of it can be found in this blog.

The next phase had me create the Tango Device Server for the hexoskin vest. This was assumed to consume a lot of time and effort, but I was able to complete this within a week. This Tango server used the helper module that was developed in the previous phase to access the data from the hexoskin vest in real-time.
A new requirement was put up, a Graphical User Interface. After a small chat with my Mentor, we decided to make the GUI using a micro-web framework called Flask, written in Python (PSF, duh!).
Meanwhile, Dipankar had begun work on creating the Anomaly Detection Algorithms. This involved a lot of research on his part, and even though he seems to hate it, I'd say he's exceptionally good at it! He really motivated me in focusing more on research and how detailed one's reports can be.
His work on the Anomaly Detection Algorithms can be found in detail in his blog.
I managed to get the GUI up and running and completed a astronaut status page and an anomaly page with the anomalies found. Everyone seemed to liked the GUI :) You can find the screenshots of it here.

One of the best learning during GSoC is how to write neat and beautiful code and how to debug. Dipankar and I really spent hours together on debugging parts of both our work.

In the final period, I managed to complete the GUI by adding in the real-time data plots as well, interactive plots. Plotly.js really helped there.
Then, my work was integrated with Dipankar's and documentations were written.

This is the main Repository of the project and documentations, here.
(Link to my GitHub)

The work during the entire summer has been explained in details through blogs(mine and Dipankar's ), two every month, for reference.

The list of merged Pull Requests can be found here.
1) Implemented Initial Utility Helper and other Resource Helpers
2) Added Hexoskin Python Client
3) Ventricular Tachycardia Integration
4) Initial Documentations and Bug Fixes 
5) Initial Tango Server Implementation
6) Web Based GUI using Python Flask
7) GUI With Plots for Detected Anomalies
8) Updated Documentation
9) Realtime Plots for Raw Data and Anomalies
10) Respiration AD Integration
11) Sleep AD Integration

I had also made a few of videos for the evaluations.

 

To conclude, I would like to thank all my three mentors and Dipankar, without whom this wouldn't have been so gracefully completed. With absolutely no regrets in entering the world of Open Source, hopefully, my journey in this field goes on.

GSoC 2017:- Final Evaluations

It was just three months back when I received the GSoC acceptance mail and here we are nearing the end of this productive summer already. This phase is the final one where we finish our projects and sign off Google Summer of Code 2017.

As mentioned in the previous post, there were a few couple of things remaining to be done in the Biometric Signal Sensor's Interface project.

The raw-data page was completely developed, which displays all the main biometrics collected in real-time. The include ECG, Respiration, Expiration, Minute Ventilation, and few more.

The raw data plots is also incorporated with the anomaly markers to indicate the corresponding anomaly caused. They are shown as inverted triangles, which are colour coded for easier recognition.

Secondly, Dipankar - my GSoC mate - was done with the final Anomaly Detection algorithms viz. Sleep Anomaly Detection and Respiration based Anomaly Detection.

This was integrated into the Tango Device Server and the Graphical User Interface was also updated to reflect these new anomaly detection algorithms.

The main part of any project, the Documentations, is present here in the form of README.md.

GSoC 2017:- Road Plan for Phase III

The second phase of the Biometric Signal Sensor's Interface, which involved writing the Tango device server for the smart shirt (Hexoskin) and developing a Graphical User Interface for the project ended neatly with a couple of merged Pull Requests.

The deliverable for the second evaluation was a video demonstrating the GUI built. The video shows the steps in running the Tango Device server and the GUI. The GUI includes a landing page which includes the details of the astronaut, basic biometrics and a status box that displays his current status. (More details here).

Phase three marks the final phase of this year's Google Summer of Code!

What's remaining?

• Add a raw data page to the GUI with real-time data displayed as plots
• Add anomaly markers on the raw data plots
• Integrate new Anomaly Detection algorithms, and
• Conclude with detailed documentation 

The first two requirements are almost complete, i.e the raw data page. The plots are drawn using Plot.ly, as done for the anomaly plots.

Another anomaly detection algorithm based on respiration is being developed and shall be integrated with the GUI and the Tango server.

Finally, documentations! The most important aspect of any project.

GSoC 2017: Second Evaluations

As laid out in my previous post, development of the Tango Device Server for the Biometric Signal Sensor's Interface and the Graphical User Interface (GUI) for the hexoskin smart shirt data collection and anomaly detection were the next milestones in line for the completion of this project.

The Tango control system is a free open source device-oriented controls toolkit for controlling any kind of hardware or software. The Tango Device server for the project is the heart of the project that integrates all the modules/components of the system. It provides various endpoints that is used for the monitoring of the astronauts.

The Tango Device Server is built by defining a server class that inherits from the Device class, using PyTango. The class definition consists of mainly two items:

• Attributes
• Commands

The attributes are certain variables that characterize the class and it's instances. The Biometric Device server has the following attributes.

• username
  Username of the astronaut | Read-only | string
• recordID
  Record ID of the current live session. Hexoskin records all the sessions of activities using record IDs | Read-only | integer
• userinfo
  Astronaut account information as a JSON string. This is consumed by the GUI to display these details | Read-only | string

 The commands are the endpoints that is opened for the clients to call, to perform actions.

• af_to_gui
  Atrial Fibrillation anomalies are retrieved from the database and provided to the client as a JSON string. This is consumed by the Flask GUI server that is the client.
• vt_to_gui
  Ventricular Tachycardia anomalies are retrieved from the database and provided to the client as a JSON string. This is consumed by the Flask GUI server that is the client. 
• apc_to_gui
  APC/PVC and PVC-hamilton anomalies are retrieved from the database and provided to the client as a JSON string. This is consumed by the Flask GUI server that is the client. 
• start_monitoring
  This is the command used to start the biometric monitoring. It starts all the three anomaly detection algorithms developed till now - Atrial Fibrillation,
  Ventricular Tachycardia and APC/PVC and PVC-hamilton - in the background. 

This created Tango Device server is then added into the database using a small register_monitor.py script. Once the device server starts running, it is available to the client. In this case, the client is the GUI.

The graphical user interface has been developed using Python Flask, and the front-end is neatened up using Materializecss. The GUI is split up into web pages:

1. The landing page :- This consists of the generic details of the astronaut such as the name, email ID, username, height, weight, etc. It also showcases basic biometrics such as Resting heart rate, BMI, VO2 max, respiration coefficient and much more.
   The major element in the landing page is the Astronaut status box. This box shows the condition of the astronaut. The box - when green - means that there has not been any detection of anomaly in the past X (say, 7) days. A red box (as shown in the picture below) depicts the state where there was an anomaly detected in the past 7 days.
   
   
   
   
2. The anomalies page :- The anomalies detected - using Dipankar's anomaly detection modules - are displayed in this page. They are displayed graphically using interactive plots, using plot.ly. But instead of having the plot made online and embedding the plots inside website using the URL, the plots are made in the Flask Server using code, and displaying it on the front-end end using Plotly.js.
   
   
   

GSoC 2017:- Road Plan for Phase II

The first phase of the Biometric Signal Sensor's Interface, which was to finish writing the modules for Hexoskin Smart Shirt polling, the health monitor helper for the Tango device server and the integration of the Anomaly Detection module is wrapped up.

First evaluation was cleared and as the deliverable to demonstrate the work done a video of the working of modules was made. The video shows the hexoskin dashboard indicating the real-time data that was collected - yes, I was wearing the Hexoskin Smart Shirt. The video progresses to display the output of the modules (raw data collected from the Smart Shirt, Atrial Fibrillation Anomaly Detection and Ventricular Tachycardia Anomaly Detection) that was developed, on the terminal.

Now, the following requirements lie -

• Tango Device Server
• Graphical User Interface (GUI) for the hexoskin data collected and anomaly detected

The TANGO control system is a free open source device-oriented controls toolkit for controlling any kind of hardware or software and building SCADA systems.
The Tango Device Server is to be developed that polls the Smart Shirt, gathers data, requests anomaly detection from the AD (Anomaly Detector) module and finally presents that to the user in a GUI.

The development for this device server requires the development of what are known as commands, that would do the above said tasks.

The GUI to display the required information is the next requirement. The GUI system should communicate using Tango exclusively, i.e it should not read directly from the Hexoskin device, using the Hexoskin API, but use the Tango device server.

A good choice for would be to use Python Flask for building a web based GUI.

As far as division of the work goes, I plan on getting the Tango device server, with necessary commands for the anomaly detection and the GUI, done first. But assuming that this wouldn't be quite exhaustive, the Flask based GUI can be soon started.

GSoC 2017: First Evaluations

My other two blogs were short and sweet. You aren't that lucky this time!
Two months into Google Summer of Code 2017 and First Evaluations being just around the corner, a detailed blog is in order.

As I've mentioned in the introductory blogs, my task, along with Dipankar, is to work on the Biometric sensor's interface for astronauts (though we have very independent roles to play in the same project). The overall project revolves in interfacing the various sensors that measure astronaut's biometrics with the Mars Society's central control system that is built using Tango Controls.

The device we chose for the astronaut (mentioned in the previous blog) took care of most of the sensors we needed.

Now it was time to get dirty with code.
We decided to have good abstraction between various components of the system.
As of now, we have the following modules:
1) anomaly_detector
2) health_monitor
3) hexoskin_helper

These three modules act as packages that the main Health Monitor Tango Device Server would use. So the work on building these three components began, keeping the Tango Device Server for later.

My work was focused on making the Health monitor package and Hexoskin Helper packages.

The hexoskin helper package solely deals with polling the smart shirt and getting the  data from various sensors on-board the shirt in real-time.
The health monitor package acts as the backbone of the system and an interface for the various components and the device server.

The anomaly detector is the pure geek stuff done by Dipankar that contains various algorithms for detecting anomalies in the astronauts' biometric data. Two algorithms has been implemented so far - atrial fibrillation and ventricular tachycardia.  This calls for method in the hexoskin helper that needs to get the desired measurements from the shirt in real-time.
The Atrial Fibrillation requires RR-Intervals and HeartRate Quality, and the Ventricular Tachycardia requires ECG, RR-Intervals,
HeartRate Quality, RR-Interval-Status and Heart Rate.

The Atrial Fibrillation interface in the health monitor - to use the methods and functions from the Anomaly Detector module - was straight forward approach, but the same for the Ventricular Tachycardia was a tricky business because of the complex flow that it follows. Different functions from the Anomaly Detector were run on different threads to get the interface ready. Details about this complex procedure can be found here.

The hexoskin helper is split into 3 parts:

• Utility helper:- That takes care all non-biometric related stuff like authentication for polling the smart shirt, accessing various records (a live session), etc. Also all the datatypes (the hexoskin biometrics) are defined along with their IDs. This makes it easier to extend the system for more sensors without having a need to dig into the hexoskin documentation, which is excellent.
  The data sample rates are also defined, in 256/samples/second, which makes sure that the data is collected properly from the hexoskin server in batches and with correct sampling rates.
  
• Resource helper:- This has all the methods that is used for the data collection - previously collected data as well real-time. Real-time data is collected using generator functions in python, that yield the biometric data every, say, 5 seconds.
  
• Database helper:- A small helper file that facilitates the database usage for storing the detected anomalies. 

The database used is SQLAlchemy (ORM - Object Relational Mapping).

Health monitor contains a a simple file that will be imported by the Tango device server. This file acts as an interface between the device server and all the components. It also contains the database where the anomalies are stored.

With the not-so-basic skeleton and the prerequisites for the device server being done, the next phase will be getting these components hooked up with a device server. Also various other anomaly detection algorithms are in queue and will be incorporated into the system.

First week as a GSoCer

Since the amazing birthday present I received last month, life as a GSoC-er has been amazing. Selected under Italian Mars Society (IMS), a sub org. under Python Software Foundation, my task for the summer is to aid missions to mars by working on a Biometric Signal Sensor's Interface.
This project is part of the current studies on the simulation of Astronaut’s Health Monitor Systems.

 When astronauts travels to Mars, or even other planets, they will be exposed to a number of hazards e.g. radiation, microbes in the spacecraft, planetary surface toxic dust. This project mainly revolves around configuring biometric signal sensors.

 The project has an initial phase of surveying the commercial sensors available, and selecting one, or many, after extensive research. The sensors are selected such that they read the biometrics like Heart Rate, Accelerometer data, Respiration rate, body temperature, Pulsioximetry data,Respiratory Volume, ECG, etc.

After the survey, it was found that Hexoskin - a wearable body metrics smart shirt fit the bill.

As their website says, it measures heart rate, HRV, breathing rate, cadence, steps, ECG, sleep positions and a lot more.

The hexoskin smart shirt is used as the device that the astronauts will primarily wear and using which their health will be monitored.

The astronaut wears the smart shirt, plugs in the hexoskin device to the shirt, sync his/her android/iOS mobile (with the hexoskin app installed) and starts exploring Mars. The application on the phone, collects the data in real-time from the shirt and sends it to their server.
This is the data we exploit to monitor the astronaut and detect any sort of anomalies that occur.
The data accessed in real-time from the shirt is sent to a Anomaly Detector - another project under IMS taken up by a friend and a fellow GSoCer - to detect various kind of anomalies, Atrial Fibrillations for starters.

The initial weeks as a GSoCer - after acquiring the device - was quite exciting, especially in the Community Bonding period. Interaction with mentors, fellow GSoC-ers and everyone else in the community has been great.

With a couple of merged pull requests, initial work around getting the data from the smart shirt has been going well.

It's time to speed up development with the first evaluations coming around in a couple of weeks.

As said before, the summer's going to be interesting!