Seqana Data Transfer App

Jakob Levin
March 18, 2021
min read

At Seqana we empower climate positive land use managers to conduct analyses of soil organic carbon (SOC) with ease and efficiency. We would like to introduce and share the development story of one of our useful internal tools that is used to ingest and process data – the Seqana data transfer app.

Soil data are central to our mission to enable data-driven carbon sequestration which is why we created this tool to integrate data from various sources into our algorithms. While there have been efforts to standardize soil data over the past decades, multiple datasets on soils with a variety of formats remain that require harmonization. This is where our data transfer app comes in handy.

To aggregate and process this data, our app easily allows users to ingest their data in various formats and allows for rapid processing. A general overview of the functionality of the app can be seen in figure 1.

Figure 1: General overview of the Seqana data transfer app’s components

To enhance user accessibility, the app was made to be lightweight, the codebase simple to read and easy to maintain, without compromising the flexibility to scale up to a complex application in the future. Since python is our main programming language at Seqana and previous experience with the web development framework, Flask, has been successful, we chose to utilize it again in developing our new tool.

The main functionality of the app allows users to upload data as files from a local or cloud storage directory (such as a shared drive or a cloud storage bucket) to a dedicated cloud storage location. While uploading, the users should provide metadata through a user input form. For users to preview their data, the app allows a quick glimpse at the data about to be uploaded through the preview functionality. Over the next sections, we want to give you some insights into how we developed these features and how you can use them in your SOC studies.

Transfer files from local/shared drives to cloud storage:

We implemented an abstract Storage class with abstract methods: list_contents, get_file, put_file. Other classes such as LocalStorage, GoogleDriveStorage, GoogleCloudStorage, GitLabStorage inherit from Storage and implement its methods based on their specific “needs”. The abstract class and methods make it easy to implement changes and thus make the code flexible. Code snippet 1 displays the original and simple implementation of the abstract Storage class and its methods, while code snippet 2 reveals the particular implementation of the GoogleCloudStorage class, inheriting from Storage, with the implementation of the put_file_binary method which can be used to upload files.

class Storage:


 def get_file_binary(self, file):



 def put_file_binary(self, file, path):



 def list_contents(self, path, nested=False):


Code snippet 1: Implementation of abstract class and methods for the Storage class.

class GoogleCloudStorage(Storage):

   # CONSTRUCTOR ------------------------------------------

 def __init__(self) …  

   # PUT --------------------------------------------------

 def put_file_binary(self, file, gc_storage_path):


     # parse path : '/data/subfolder/file.txt'= ['data','subfolder','file.txt']

     gc_storage_path_parts = Path(gc_storage_path).parts

     # Get the bucket on google cloud storage to where to upload the file

     gc_bucket = self.__get_connection().get_bucket(gc_storage_path_parts[0])"Got destination Bucket")

     # create a "file-holder" on the cloud storage in the specified bucket

     blob = gc_bucket.blob(




             )  # create path from the array of the parts

         )  # transform it into a string


     # the file is uploaded via stream without downloading it to the local storage

     blob._CHUNK_SIZE_MULTIPLE = 5 * 1024 * 1024

     blob._chunk_size = 5 * 1024 * 1024


         file, rewind=True

     )  # rewind=True -> bring the pointer to the beginning"Uploading file to gcs is done")

     # returns a public url to the file

     return blob.public_url

Code snippet 2: Implementation the GoogleCloudStorage class with put_file_binary method.

A wrapper class StorageManager was developed which manages the transfer of data files from source to destination. This structure allows the movement of files between two storage classes, irrespective of their specific type.

Now, you can simply initiate the instances of the storage types, specify the source and destination file paths to be moved, and send this information to the StorageManager which handles the rest for you. An example script is shown in code snippet 3.

# 1. Init google drive storage

google_drive_storage = GoogleDriveStorage(cquest_metadata.gdrive_secret_payload)

# 2. Init google cloud storage

google_cloud_storage = GoogleCloudStorage(cquest_metadata.gcp_access_creds)

# 3. Init storage manager

storage_manager = StorageManager()

# 4. Move files






     cquest_metadata.gcp_bucket_name, result_dict["short_title"][0]



Code snippet 3: Example for moving data between google drive and a cloud storage bucket

User Input Form:

The user input form to upload data comes in the form of a questionnaire which is displayed in figure 2. This questionnaire is implemented through a JSON file, which is retrieved by calling the method get_file_binary().

Figure 2: Screenshot of the user input form of our app

Then the file is passed to the questionnaire.html with flask and the command:

render_template("questionnaire.html", table=json.dumps(questionnaire_json))

JavaScript then parses the JSON file and dynamically generates HTML content for the questionnaire page. While doing this JavaScript completely relies on the information passed to it from the JSON object. For example, the “question_type” field defines what type of input the HTML has to add (text, radio, checkbox, date, ect. ), “question_string” defines how the question should be presented on the web page, “question_options” define the available options from which the user can pick the answer, etc. This design enables addition or modification of the questionnaire simply by changing the corresponding json file without touching the code or redeploying the app.

Preview File:

To increase assurance of accurate work, we developed the functionality to preview file content before uploading. This is implemented with JavaScript: when the input file gets selected an eventListener triggers an execution of a function. This function converts the binary of the selected file into a json object and then dynamically generates an HTML table content for the page. Check out this last code snippet 4 for more details:

// FILE PREVIEWER : handle

function handleFilePreview(e) {

 // Adds Names of the files as a list

 if(! return;

 prevFileDiv.innerHTML = "";

 // get file

 var file =[0];

 var fileType =".")[1]

 // read file

 var fileReader = new FileReader();

 if(fileType == "csv"){


 } else if (fileType == "xls" || fileType == "xlsx") {



 // load file to fileReader and generate html string for inner content

 fileReader.onload = function() {

     var htmlString = ""

     const text = fileReader.result;

     var jsonObject = null;

     // convert csv to json object

     if(fileType == "csv"){

         jsonObject = csvJSON(text); // separately implemented function

     // convert xlsx file binary to json object

     } else if (fileType == "xls" || fileType == "xlsx") {

         jsonObject = xlsJSON(text); // separately implemented function


     const keys = Object.keys(jsonObject[0])

     // create table

     htmlString += "<table class='table table-striped'>";   // <-------- START TABLE

         // code for dynamical html content generation        

     htmlString += "</table>"   // <-------------------------------------- END TABLE

     prevFileDiv.innerHTML = htmlString;



Code snippet 3: Example for moving data between google drive and a cloud storage bucket

As we developed the Seqana data transfer app to assist users with studies and analyses of SOC, we are excited for the opportunity for you to utilize this tool. At Seqana, we strive to make SOC projects as simple, efficient, and organized as possible in order to make efforts to combat the issue of atmospheric CO2 accessible to anyone interested. So, for the time being, we develop and use this app as an internal tool, but we are open to share access and let you use the app together with us. Please reach out if you are interested or have any questions.

Soil Organic Carbon
Seqana is an EU-funded, earth observation and ML enabled, SaaS company that is developing credible and cost-efficient soil carbon monitoring, reporting, and verification (MRV) tools for regenerative agriculture. We enable change-makers to combat the effects of climate change by introducing transparent, replicable, and scalable MRV into this nascent market.
© 2024 Seqana. All right reserved.