DOCS InstallGet StartedUser GuideRelease Notes

Using ObjectiveFS for Uploading and Processing Incoming Data

This article describes an example workflow that uses ObjectiveFS to upload and process incoming data. In this workflow, the server that uploads data is different from the server that processes data and both mount the same shared filesystem.

Setup

  • Server A: uploads data to filesystem
  • Server B: processes data on filesystem after data has been uploaded
  • An ObjectiveFS filesystem is mounted on both servers at /ofs. This filesystem contains 3 directories:
    • upload: the directory where new files are uploaded
    • process: the directory where files are processed
    • done: the directory where files are moved to after processing

Steps

Server A uploads files to the /ofs/upload directory.
Once a file upload is complete, server A moves that file into the /ofs/process directory.
Separately, server B checks if new files are present in its /ofs/process and processes these files. Once the processing has completed, server B moves the files to the /ofs/done directory. An example bash script for server B is shown below.
#!/bin/bash
while true; do
for f in /ofs/process/*; do
    process $f
    mv $f /ofs/done
done
sleep 1
done

Last updated ObjectiveFS staff

ObjectiveFS is a shared file system for Linux and macOS that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com