Maintaining data integrity with unstable connections

We want to add offline capabilities to our angular app. That means when the user is offline and needs to update/add something to the server via HTTP call, the app will save the call(we just named it UpdateObject) in the device local storage.

The local storage will save an array of UpdateObject And we will be calling it the UpdateQueue. When the app goes online, HTTP calls will be made for each UpdateObject.

interface UpdateObject {
  url: string;
  method: string; // GET, POST, PUT, DELETE
  payload?: any;
}

This is a Update queue

UpdateObject[]

updateCount is a rxjs Subject. We will subscribe it and listen for changes in the queue.
updateCount emits value whenever the number of objects changes inside queue.

updateCount: Subject

So if it is an Angular service, the start looks like this

interface UpdateObject {
  url: string;
  method: string; // GET, POST, PUT, DELETE
  payload?: any;
}

@Injectable()
export class UpdateQueueProvider {
  public updateCount: Subject; /* number of updateObjects in updateQueue, it's a BehaviorSubject */
  public updateQueue: UpdateObject[]; 
  private updateQueueSub: Subscription; 
  public updateQueueFailed = []; 
  private tryCount: number = 0;

Let’s write a method for adding an object inside queue

  public addToQueue(obj: UpdateObject) {

    this.storage.get('updateQueue')
      .then(value => {
        let queue = value || [];

        // Update queue and save to storage
        queue.push(obj)
        this.storage.set('updateQueue', queue);
        this.updateQueue = queue;
        // Trigger event;
        this.updateCount.next(this.updateQueue.length);
      })
  }

First, we check if we have an existing queue in device/browser store. If not, we initiated an empty array then push UpdateObject to Queue, saved the queue to store. Finally, we called next method on updateCount.

For storage, you can use anything you want like LocatStorage of the browser. In our case, we used ionic storage.

Now let’s take a look at the ‘init()’ method which actually does the main job.

  private init() {
    let uq = this.storage.get('updateQueue');
    let uqf = this.storage.get('updateQueueFailed');
    Promise.all([uq, uqf])
      .then(values => {
        this.updateQueue = values[0] || [];
        this.updateQueueFailed = values[1] || [];
        this.updateCount = new BehaviorSubject(this.updateQueue.length);
        // Listen for updateCount change
        this.updateCount.subscribe(res => {
          // console.info('updateCount.subscribe', res)
          // Check connectivity
          if (!navigator.onLine || !this.updateQueue.length) {
            return;
          }

          // Prevent registering multiple listeners of API call
          if (this.updateQueueSub) {
            this.updateQueueSub.unsubscribe();
          }

          // API CALL - Save first item from Queue
          let upddateObj = this.updateQueue[0];
          this.updateQueueSub = this.saveToRemoteDB(upddateObj)
            .subscribe(data => {
              console.log('Saved to remote DB - response', data);

              // Success - Remove first item and Update localstorage
              this.updateQueue.shift();
              this.storage.set('updateQueue', this.updateQueue)

              // If Queue has item then trigger event again
              if (this.updateQueue.length) {
                setTimeout(() => {
                  this.updateCount.next(this.updateQueue.length - 1)
                }, 2000)
              }
            }, err => {
              // On error - Remote DB save
              // try 3 times
              console.log('error posting:')
              console.log(err);
              if (this.tryCount <= 2) {
                setTimeout(() => {
                  this.tryCount++;
                  this.updateCount.next(this.updateQueue.length - 1);
                  console.log('Failed and try ', this.tryCount);
                }, 2000)
              } else {
                this.tryCount = 0;
                // Failed - Remove first item and save both failed and main queue
                let failedObj = this.updateQueue.shift();
                this.updateQueueFailed.push(failedObj);
                this.storage.set('updateQueueFailed', this.updateQueueFailed)
                this.storage.set('updateQueue', this.updateQueue)
                console.log('Failed and try next item in queue');
                if (this.updateQueue.length)
                  this.updateCount.next(this.updateQueue.length - 1)
              }
            })
        })
      })
  }

‘storage.get()’ returns Promise. So we are getting updateQueue and updateQueueFailed from two Promises by calling ‘Promise.all()’. updateCount is initiated with the value of updateQueue.length. And immediately after that, we subscribed to updateCount. Inside callback, if the device is online and the queue has updateObject we get the first updateObject from the queue.

Then we made HTTP request by calling ‘saveToRemoteDB(upddateObj)’ method and passed the updateObject to this method. This method returns observable. So we subscribed and if the HTTP request is made successfully we removed the first updateObject from the queue and saved the queue to device/browser storage.

Then if the queue has object we called the “next()” method from updateCount to start this cycle again.

Inside error callback of HTTP request we sent the same request 3 times. If the requests still fails, then we save the failed updateQueue to storage for later inspection.
And finally we call “next()” from updateCount and start the cycle again.

We added an event listener to network connectivity. When app is online it starts the update cycle if the queue has any updateObjects.

onAppOnline() {
    window.addEventListener('online', () => {

      this.storage.get('updateQueue')
        .then(value => {
          this.updateQueue = value || [];
          this.updateCount.next(this.updateQueue.length);
        })
    });
  }