Creating a High-Performance Image Resizer Proxy as a Dockerized Go Microservice

While working on applications that process a large number of images, we often come across the requirement to resize images for different resolutions. An image generated by a typical 12MP mobile phone camera can be 5MB-12MB depending on the quality of the image. Once uploaded, the image must be optimized for different devices. As you know, image resizing is a complex mathematical process. The performance and quality of the optimized image really depend on the efficiency of the algorithms used. 

Let me walk you through a high-performance solution for image resizing. We will implement the solution using libvips, a fast open-source image processing library, along with Golang and expose the resize functionality via a proxy server. With even 20% resource usage improvement per image, it can make a world of difference when scaling up to millions.

Note: This is not an attempt to compete with existing libraries. Our aim is to build our own proxy server to handle custom resizing URLs or custom pre/post-processing of the resized images within the same proxy layer. As the code is our own and has minimal overhead, we can tweak it dynamically to match any requirement without calling a second microservice.

What we will do:

• Implement a proxy layer in Golang.
• Modify the proxy code to intercept the actual response data and return the resized data.
• Process or save the resized image in S3 parallely without blocking the response.
• Dockerize our proxy layer for easy scaling as well as managing dependencies.
• Prepare the Docker image with necessary dependencies (libvips) for image resizing. 

By the end of this article, you will have a working Docker-based image resizer that you can replicate easily on your local machine. The source code is provided at the end of this article.

Why write a custom proxy?

There are libraries in different languages for image resizing that cater to requirements like matching an existing image resizing service’s URL formats, combining many operations on the image, and supporting multiple storage and caching layers. But using a generic library can often bloat the microservice and impact performance as well as maintenance overhead. Here we are using Golang and libvips for our custom implementation.

Golang’s net/http package comes with most of the implementation required for client-server communication. Using standard http libraries, you can easily create a proxy. Go’s performance and memory safety also make it a very good choice in our case.

Libvips is one of the fastest open-source libraries for image resizing written in C. It needs a low memory footprint and is nearly four times faster than the quickest ImageMagick and GraphicsMagick configurations or Go native image package. While processing jpeg images, it can be five to eight times faster. Here we will use Bimg, a Golang wrapper for libvips inside the proxy code.

Let's start with the proxy code

Using the below code, we will create a simple proxy service in Golang listening on port 8080.

In the main function, we will create a new proxy object, which will take our incoming request and relay it to the target image server via our proxy server. 

Next, we will define the transport struct for handling all the transports occurring via our proxy. We will implement the RoundTrip method for the transport struct, which will allow us to modify, throttle, or even reject the original requests with different response codes. This code will hit the target (image) server with the same url and return the response. We can optionally process or modify the response data  before sending it back to the original caller as discussed in the next section.

Let's add our resizing code to the proxy

We will now modify the proxies RoundTrip method to perform the following actions in a sequence. (Please refer to the numbered comments added inline.)

1. Make a RoundTrip request with the original URL and serve back if the image exists.
2. If the requested URL is not found, try to extract the width and full-res filename.
3. If the width is available, extract the original file URL and modify  the request with the newly extracted original image file URL.
4. Use the modified request object to make a  RoundTrip request, which will fetch us the image.
5. If the original image is found, resize it using resizeImage method which updates the response body before returning, else returns the response as such.

The resizeImage() method does the following operations:

1. Receives the S3 upload path (for caching back), the originalImage file response, and the resize width.
2. Reads the response body into memory and stores it in a bytes array.
3. Creates a new image by passing the image bytes using bimg.
4. Calculates the relative height using aspect ratio calculation.
5. Applies resize operation on image object using bimg methods by passing width and height.
6. Modifies the  response body by replacing with new bytes[] from the resized image’s bytes array.
7. Alters the Content-Length (and Content-Type ) headers of the response based on the new data.
8. Does a background processing of the resized image bytes, like uploading back to S3 or the original image server. (The storage package is not included in the post for the sake of brevity.)

Setting up Dockerfile

Next, we will set up the environment using Docker.  Libvips and dependent libraries need to be installed so that we can call libvips via the Go wrapper. We will be using a Docker-based setup in order to avoid dealing with installations and unexpected system dependencies.

We will use an Alpine Linux image for our Docker so as to keep a minimal footprint from the base OS image. Alpine has the libvips dev package precompiled.

To prepare the Docker image, install the libvips precompiled dev dependencies from alpine libs. Next, install the build-base in order to get gcc and the required build tools.

Copy the Go source code onto the Docker work directory and then compile it into an executable. Finally,  expose the port where the go-proxy server will listen to. Then run the executable we compiled in the previous step.

(For production use cases, we may need to freeze a specific version of libvips. This can be done by downloading and compiling the source code in the build steps.)

The Dockerfile content is shown below:

Downloading and running the project

We will use a docker-compose file to set up our project. We will use MinIO Docker to replicate the image server.  The compose file will have two services defined in the same network, one proxy service [:8080] and one MinIO s3 service [:9000].

To extract the resizer microservice alone, without the image server, remove the MinIO service from the docker-compose file.

Check out the project on GitHub.

There is a run.sh file provided at the root of the project. Open the root folder in the terminal and then you can quickly run it from the command prompt using
./run.sh

If everything goes well, you can see the following logs with two containers running. 

Testing the microservice

Once the dockerized proxy is up, we can test our service using the same URLs that we used to access the original images. Just replace the domain/server part with the new proxy server address.

You can access the MinIO S3 GUI  @ localhost:9000 using default credentials minioadmin:minioadmin and upload your own files. A few images are provided for testing in the images bucket.

Suppose our image server is at [minio:9000], we will add this server to the PROXIED_SERVER environment variable and the URLs would be changed from:

http://minio:9000/images/logos.png  → http://localhost:8080/images/logos.png

For resizing, to say 640px width, we will now call the same URL by appending [_width] with the original URL:

http://localhost:8080/images/logos.png_640

Simply put, you can access the original files on the S3 image server directly by using

http://localhost:9000/images/logos.png

And the proxy resized version using

http://localhost:8080/images/logos.png_640

Some examples

Let's make a single request to get our logos.png image resized with a 640px resolution. The original image already exists at http://minio:9000/images/logos.png.

The client will need to have only two minor changes in the URL:

1. The domain/servername part.
2. Appending the required width using _width.

Hit our proxy by:  http://localhost:8080/images/logos.png_640

And that's it! You will have your 640px logo served without consuming a separate API or using any external services.

From the below logs, you can see the operations done on the proxy server.

Now, if you make multiple requests to fetch the same resolution, the stored image will be sent back without undergoing any resize operation.

Lets try a new resolution of 800px  for logos.png:

http://localhost:8080/images/logos.png_800

As you can see from the logs and screenshots below, subsequent requests do not need any resize operations if we implement background upload of a resized image on the first hit. This keeps the code dynamic and takes care of any non-existent resolutions automatically.

Further improvements

You can explore the source code and try out various improvements such as:

• Adding a height option to the url.
• Adding response format option, webp/avif etc. to reduce image sizes even further.
• Watermarking your images using libvips watermark function.
• Adding a thumbnail option to the url by making use of smart vipsthumbnail functions.
• Turning the microservice into a lambda to further reduce the cloud costs.
• Limiting the background upload for only common resolutions.
• Writing your own custom storage adaptors for uploading resized images.
• Exposing an API-based resize option that can be used to do bulk image resize during initial content creation or a batch image resize and upload.
• Throttling maximum number of requests (for example, uber rate-limit) as image resize is CPU- and memory- intensive.
• Testing and modifying the proxy to serve https requests.

Performance Comparison

We used our solution in one of our projects to offload an NGINX-based server from image resizing tasks.  Doing image resize operations using the NGINX image-filter plugin alongside normal request handling caused other requests to be delayed as the resize operations occupied most of the CPU time. Using libvips reduced the processing time by more than half and freed up the NGINX server containers from peak loads during bulk resize requests. Also caching the resized images back to S3 eliminated any repeated resize requests.

Here are a few readings from the project, which was run on a development machine with i7 16gb ram.

Image1: 5400x6830pixels jpg - 6.5MB 

Width        nginx    go-libvips
1300px     -    4.1s    2s
240px      -    3.2s    1.2s
32px         -    2.8s    1.1s

Image2: 3806x2855pixels jpg - 1.8MB

Width        nginx    go-libvips
1200px     -    1.12ms    360ms
800px     -    960ms     425ms
32px     -    680ms     210ms

A detailed performance study of various image libraries, including pure Go solutions, is available in this GitHub repository. You can pull the project and run the tests to check how different libraries perform in comparison to each other.