Often there’s a need to abstract away your machine learning model details and just deploy or integrate it with easy to use API endpoints. For eg., We can provide a URL endpoint using which anyone can make a POST request and they would get a JSON response of what the model has inferred without having to worry about its technicalities.

In this tutorial, we will create a TensorFlow Serving server to deploy our InceptionV3 image classification convolutional neural network (CNN) built in Keras. We will then create a simple Flask server which will accept POST request and do some image preprocessing, required for Tensorflow serving server, and return a JSON response.

What is TensorFlow Serving?

Serving is how you apply machine learning model after you’ve trained it.

TensorFlow Serving makes the process of taking a model into production easier and faster. It allows you to safely deploy new models and run experiments while keeping the same server architecture and APIs. Out of the box, it provides integration with TensorFlow, but it can be extended to serve other types of models.

Installing TensorFlow Serving

Prerequisite: Please create a python virtual environment and install Keras with TensorFlow backend in it. Read more here.

Note: All the commands have been executed in python virtual environment on Ubuntu 18.04.1 LTS.

Now, inside the same virtual environment run the following commands (use sudo for root permissions):

$ apt install curl$ echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | sudo apt-key add -$ apt-get update$ apt-get install tensorflow-model-server$ tensorflow_model_server --version
TensorFlow ModelServer: 1.10.0-dev
TensorFlow Library: 1.11.0$ python  --version
Python 3.6.6

You can upgrade to a newer version of tensorflow-model-server with:

$ apt-get upgrade tensorflow-model-server

Directory overview of what we are going to build

Understanding directory structure, before we get started, will help us in getting a clear picture of where we are at each step.

(tensorflow) ubuntu@Himanshu:~/Desktop/Medium/keras-and-tensorflow-serving$ tree -c
└── keras-and-tensorflow-serving
    ├── README.md
    ├── my_image_classifier
    │   └── 1
    │       ├── saved_model.pb
    │       └── variables
    │           ├── variables.data-00000-of-00001
    │           └── variables.index
    ├── test_images
    │   ├── car.jpg
    │   └── car.png
    ├── flask_server
    │   ├── app.py
    │   ├── flask_sample_request.py
    └── scripts
        ├── download_inceptionv3_model.py
        ├── inception.h5
        ├── auto_cmd.py
        ├── export_saved_model.py
        ├── imagenet_class_index.json
        └── serving_sample_request.py6 directories, 15 files

You can get all of these files from my GitHub repository:

Exporting Keras model for Tensorflow Serving

For this tutorial, we will download and save InceptionV3 CNN, having Imagenet weights, in Keras using download_inceptionv3_model.py. You can download any other model available in keras.applications library (here) or if you have built your own model in Keras then you can skip this step.

After executing the above script you should get the following output:

$ python download_inceptionv3_model.py
Using TensorFlow backend.
Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.5/inception_v3_weights_tf_dim_ordering_tf_kernels.h5
96116736/96112376 [==============================] - 161s 2us/step

Now we have our InceptionV3 CNN (inception.h5) saved in Keras format. We want to export our model in a format that the TensorFlow server can handle. We do this by executing export_saved_model.py script.

TensorFlow provides the SavedModel format as a universal format for exporting models. Under the hood, our Keras model is fully specified in terms of TensorFlow objects, so we can export it just fine using Tensorflow methods. TensorFlow provides a convenience function tf.saved_model.simple_save() which abstracts away some of these details and works fine for most use cases.

Output:

$ python export_saved_model.py
WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

We get this warning because we have downloaded a pre-trained model. We can use this model for inference as is, but if we want to train it further, we need to run the compile() function after loading it. This warning can be safely ignored for now. After executing this script, the following files are saved in my_image_classifier directory:

├── my_image_classifier
   └── 1
       ├── saved_model.pb
       └── variables
           ├── variables.data-00000-of-00001
           └── variables.index2 directories, 3 files

Suppose we want to update our model in the future (maybe because we have collected more training data and trained the model on the updated dataset), we can do so by,

1. Running the same script on the new keras model
2. Updating export_path = ‘../my_image_classifier/1’ to export_path = ‘../my_image_classifier/2’ in export_saved_model.py

TensorFlow Serving will automatically detect the new version of the model, in my_image_classifier directory, and update it in the server.

Starting TensorFlow Serving server

To start TensorFlow Serving server on your local machine, run the following command:

$ tensorflow_model_server --model_base_path=/home/ubuntu/Desktop/Medium/keras-and-tensorflow-serving/my_image_classifier --rest_api_port=9000 --model_name=ImageClassifier

• --model_base_path: This has to be an absolute path else you will get an error saying:

Failed to start server. Error: Invalid argument: Expected model ImageClassifier to have an absolute path or URI; got base_path()=./my_image_classifier

• --rest_api_port: Tensorflow Serving will start a gRPC ModelServer on port 8500 and the REST API will be available on port 9000.
• --model_name: This will be the name of your Serving server using which you will send a POST request. You can type any name you want here.

Testing our TensorFlow Serving server

The serving_sample_request.py script makes a POST request to the TensorFlow Serving server. The input image is passed via command line argument.

Output:

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com

$ python serving_sample_request.py -i ../test_images/car.png
Using TensorFlow backend.
[["n04285008", "sports_car", 0.998414], ["n04037443", "racer", 0.00140099], ["n03459775", "grille", 0.000160794], ["n02974003", "car_wheel", 9.57862e-06], ["n03100240", "convertible", 6.01581e-06]]

TensorFlow Serving server takes slightly more time for responding to the first request as compared to subsequent calls.

Why do we need a Flask server?

As we can see, we have performed some image preprocessing steps in serving_sample_request.py (frontend caller). Following are the reasons to create Flask server on top of TensorFlow serving server:

• When we are providing our API endpoint to frontend team we need to ensure that we don’t overwhelm them with preprocessing technicalities.
• We might not always have a Python backend server (eg. Node.js server) so using numpy and keras libraries, for preprocessing, might be a pain.
• If we are planning to serve multiple models then we will have to create multiple TensorFlow Serving servers and will have to add new URLs to our frontend code. But our Flask server would keep the domain URL same and we only need to add a new route (a function).
• Providing subscription-based access, exception handling and other tasks can be carried out in the Flask app.

What we are trying to do is eliminate tight coupling between TensorFlow Serving servers and our Frontend.

For this tutorial, we will create a Flask server on the same machine and in the same virtual environment as that of TensorFlow Serving and make use of the installed libraries. Ideally, both should be running on separate machines because a higher number of requests would cause the Flask server to slow down because of the image preprocessing being carried out. Also, a single Flask server might not be sufficient if the number of requests is really high. We may also need a queuing system if we have multiple frontend callers. Nonetheless, we can use this method to develop a satisfactory proof of concept.

Creating a Flask server

Prerequisite: Install Flask, in python virtual environment from here.

We just need a single app.py file in order to create our Flask server.

Go to the directory where you have saved your app.py file and start the Flask server with the following command:

$ export FLASK_ENV=development && flask run --host=0.0.0.0

• FLASK_ENV=development: This enables debug mode which basically gives you complete error logs. Don’t use this in a production environment.
• The flask run command automatically executes the app.py file in the current directory.
• --host=0.0.0.0: This enables you to make requests, to the Flask server, from any other machine. To make a request from a different machine, you will have to specify the IP address of the machine where the Flask server is running in place of localhost.

Output:

* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 1xx-xxx-xx4
Using TensorFlow backend.

Start the TensorFlow Serving server using the same previous command:

$ tensorflow_model_server --model_base_path=/home/ubuntu/Desktop/Medium/keras-and-tensorflow-serving/my_image_classifier --rest_api_port=9000 --model_name=ImageClassifier

Here’s a script (auto_cmd.py) to automate starting and stopping of the two servers (TensorFlow Serving and Flask). You can modify this script for more than two servers as well.

Remember to change the path at line 10 of auto_cmd.py to make it point to your app.py’s directory. You may also need to change line 6 in order to make it point to your virtual environment’s bin. You can then execute the above script from any directory by executing following in your terminal:

$ python auto_cmd.py

Testing our Flask server and TensorFlow Serving server

We make a sample request using the flask_sample_request.py script. The script basically mimics request from the frontend:

1. We take an input image, encode it to base64 format and send it to our Flask server using POST request.
2. Flask server decodes this base64 image and pre-processes it for our TensorFlow Serving server.
3. Flask server then makes a POST request to our TensorFlow serving server and decodes the response.
4. The decoded response is formatted and sent back to the frontend.

Output:

$ python flask_sample_request.py -i ../test_images/car.png
[
  [
    "n04285008", 
    "sports_car", 
    0.998414
  ], 
  [
    "n04037443", 
    "racer", 
    0.00140099
  ], 
  [
    "n03459775", 
    "grille", 
    0.000160794
  ], 
  [
    "n02974003", 
    "car_wheel", 
    9.57862e-06
  ], 
  [
    "n03100240", 
    "convertible", 
    6.01581e-06
  ]
]

Our flask server currently has only a single route for our single Tensorflow Serving server. We can serve multiple models by creating multiple Tensorflow serving servers on different or same machine. For that we just need to add more routes (functions) to our app.py file and perform required model specific pre-processing in it. We can give these routes to our frontend team to call the models as required.

Handling Cross-Origin HTTP request

Consider a scenario where we make a POST request using Angular, our Flask server receives OPTIONS header and not POST because,

• A web application makes a cross-origin HTTP request when it requests a resource that has a different origin (domain, protocol, and port) than its own origin.
• CORS (Cross Origin Resource Sharing) is a mechanism that uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin. Read more about CORS here.

Hence, Angular doesn’t get back any response from the Flask server. To solve this we have to enable Flask-CORS in our app.py. Know more about it here.

Conclusion

And that’s all we need to serve our machine learning model. TensorFlow Serving makes it really easy to integrate machine learning into websites and other applications. With plenty of prebuilt models available in keras (here), it’s possible to develop super useful applications with minimal knowledge of machine learning and deep learning algorithms.

If you found this tutorial helpful, please do share it with your friends and leave a clap :-). If you have any queries, feedback or suggestions do let me know in the comments. Also, you can connect with me on Twitter and LinkedIn. There is so much to share with all of you and I’m just getting started. Stay tuned for more!

[출처] https://towardsdatascience.com/deploying-keras-models-using-tensorflow-serving-and-flask-508ba00f1037