Streaming requests

What do you do if you host an app in the cloud and you must process user file uploads but cannot persist (at least forever) data locally? You have a few choices:

• save in a temporary file and use a job to move the files to a safer location
• save in a temporary file, and then upload the content of this file to a safer location (while the client wait)
• stream the incoming request directly into the safe location

Pros and cons

Let’s see the pros and cons of the different approaches.

The first one is faily simple to implement. Most web application framework, atleast today, support file uploading out of the box. You save the file locally and you set up a scheduled job. The job can live in your service or in cron. But what happens if your app fails after the user upload, but before the scheduled job processed the file. You loose it.

The second one is also easy to implement. You do not loose the file if the application fails because the confirmation the file was uploaded is not sent to the client until the upload to the safe location is complete. The problem with this approach is that the client has to wait a much longer time. The file must be uploaded to you app and then to the safe location. If you are lucky, you have great bandwidth between your server and the safe location, but that is not always the case.

The last approach is the best of both world. If the app crashes, no file is lost as the request was never fully processed. The client do not wait twice the upload time because the data is sent to the safe place as it comes into the web application. The problem is that this approach can be fairly complex to implement.

How to stream

Personnally, I never had the chance to try request streaming with Spring Boot, but I suppose it is possible. I could try it in the Play Framework. I tried to do it in the 2.4 version but it took me way too much time to implement it for the benefit gained. I gave up. When Play 2.5 was released, I updated my application and gave it another try.

This time I was successful and I’m very impressed by the little amount of code you need to do it. I uploaded an example of this here, so you can try it yourselves.

This example is very basic. A real world application of this is a little bit more complex, but you should get the gist of it.