Product Hatching Ideas

DIG Upload Process Refresh

The current process for uploading files from LNK-DEB-DIG-001 is a leviathan monstrosity built in shell. It's very clearly case of a shell script trying to do a real applications job, as it features attempts to implement daemonization, multithreading, file locking, and network traffic management in shell. All of the above are at an approximate 50% buttock level, and the program is prone to responding to network disruptions by hanging.

This contributed seriously to the service disruption on 10/10/2016 where some network blips between lnk-deb-dig-001 and proc01 caused the process to hang multiple times and need human intervention to get it running again.

This upload process could be enhanced significantly with one of two strategies

1. Stop trying to reinvent the wheel. Use rsync instead. Some of the other bits this app does such as file renaming to protect work-in-process files would have to be re-implemented on the proc01 end.
2. Re-write the thing to accomplish the same effects, but use ruby/python/perl/go/c/java instead and have a proper daemonized process with proper multithreading and proper fault tolerance.

  • Krista Odbert
  • Oct 25 2016
  • Shipped
Workaround Present Yes
Impact to Existing Customers All
  • Attach files
  • Admin
    Krista Odbert commented
    October 25, 2016 18:12

    Matthew Theis added a comment - Oct 13, 2016 08:44 AM - edited
    Requirements from a DIG perspective are basically just some form of logic integrated into the code that will enable .rte files to continue to upload. Further, should the process become hung up, we would also prefer to receive some alert. At the present time, we rely on our customers and/or employees to notice that rates are not posting in a timely manner.

    The solution to the problem at the time of the incident was to direct the pipe from public to private.


  • Rob Smith commented
    January 12, 2017 15:39

    Currently we have two folders "upload & upload_v2" that our files (see attached file) dump into.  Back in the day they had to keep Wells SRPs files and .rte files in separate folders because when SRPs were updating it would slow down the process to to update the .rte files.  The reason behind having two different upload folders really don't matter and dumping all our files under one folder should not slow any of the other processes down. For consistency we need to update this process and put everything under one folder.


    Currently DiG can upload all files to test except for Prowork, and Loan Limit files.  We run the file but MOPs has to push the .txt and .pro files to the test server for us.  I talked to Nick and he did not see any reason why we could not do that ourselves.  The process could be set up the same way it is for the rest of the files we send to the upload folder.  We also need to add more of these files to the Run File report on the Admin site so we now if they process or not.  

  • Rob Smith commented
    January 12, 2017 19:32

    Rob Smith...Along with my other comment we need to have more error log reporting so we know when things break.  Emails, slack inegration etc....