Sending binary data along with a REST API request

 

The problem I would like to discuss is an API call, where you need to send binary data (for example multiple images) and some metadata information together. There are various ways you can approach this, and I will describe them briefly. Then I will go into more detail on multipart/form-data requests and how they can help you with the mentioned task.

Approach 1 – Send metadata and files in separate requests

The steps could be this:

  1. Send metadata to server
  2. Server stores metadata and generates an unique URL, to which files should be uploaded. Sends the URL in response.
  3. Client posts files to specified URL

This may be a good approach in a scenario, where you don’t need to receive the files right away together with the metadata. This enables the client to upload some initial files, then later add some more. This could be a good approach if you are creating a new photo album (metadata), then adding photos to it.

Approach 2 – Send metadata and files together in one request

There are some cases however when metadata and files are one entity and neither makes sense on its own. In this scenario you want to take some action on the whole client request data right away. Let’s say you are creating a service that combines some images into one, calculates some statistics, etc. If you used approach 1 in this scenario, you need to manage state between client requests. This can be complex and may hurt your scalability.

Fortunately you can send all data in one request if this is what makes the most sense. Here are a few ways to do this:

JSON / XML

Probably the easiest and most compatible way to send the data is to serialize it to JSON or XML. Also the binary data, which means getting +33% in message size due to BASE64 compression. This may be just fine in some cases.

 

BSON

However if the extra message size is not something you can put up with, then you can use a form of binary serialization. With ASP.NET WebAPI 2.1 you’ll get BsonMediaTypeFormatter out of the box, which will help you use BSON serialization (see http://bsonspec.org/). On the client side, you can use JSON.NET to do the serialization of the request message.

With some effort you should also be able to use Google’s protobuf protocol, which may be more efficient according to this blog post.

multipart/form-data

In some cases, maybe for compatibility reasons, you’ll not be able to use modern binary serialization like BSON or protobuf. In those cases you can still avoid sending binary data in BASE64 encoded string. You can use multipart/form-data request, effectively simulating HTML forms with file uploads behavior. This is a bit more complex than the previous approaches, so I would like to go into more detail.

But before that, I should mention, that this approach is not semantically correct. By using the Content-Type multipart/form-data you state, that what you send is actually a form. But it is not. It is rather some custom data format and for that, the most appropriate content type seems to be multipart/mixed (see the RFC). The HttpClient library for .NET won’t have any problems handling different subtypes of multipart/* MIME type, but for other platforms, it may not be true. I have seen some libraries (for Python for example), which had multipart/form-data content type hardcoded. So in this case you have two options: either write your own client library or give up being semantically correct and go with multipart/form-data. I can live with the latter.

So how to put some metadata together with multiple files into one request? Look at this example:


 

POST http://127.0.0.1:53908/api/send HTTP/1.1
Content-Type: multipart/form-data; boundary="01ead4a5-7a67-4703-ad02-589886e00923"
Host: 127.0.0.1:53908
Content-Length: 707419

--01ead4a5-7a67-4703-ad02-589886e00923
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name=imageset

{"name":"Model"}
--01ead4a5-7a67-4703-ad02-589886e00923
Content-Type: image/jpeg
Content-Disposition: form-data; name=image0; filename=Small-Talk-image.jpg


...image content...
--01ead4a5-7a67-4703-ad02-589886e00923
Content-Type: image/jpeg
Content-Disposition: form-data; name=image2; filename=url.jpg



...image content...
--01ead4a5-7a67-4703-ad02-589886e00923--

Going from the top we have a part with name = imageset. This part contains the metadata. It has JSON content type and a serialized JSON object. Then there are two parts containing image data with names image0 and image1. Those two additionally specify: image filename and image type (in Content-Type header).

The server, after receiving such request, can distinguish metadata and image data by looking at the part names (the names are  part of the API contract client needs to follow). Then it can put the request together and execute a controller action passing in received data.

So how to actually implement this using ASP.NET WebAPI? (you can look at the code at Github)

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com

Let’s start quickly with the definition of the data passed around:

    public class ImageSet
    {
        public string Name { get; set; }

        public List<Image> Images { get; set; }
    }

    public class Image
    {
        public string FileName { get; set; }

        public string MimeType { get; set; }

        public byte[] ImageData { get; set; }
    }

Now, when implementing a WebAPI controller, we would like to receive and instance of the ImageSet as a parameter of the action.

[Route("api/send")]
public string UploadImageSet(ImageSet model)
{
    var sb = new StringBuilder();
            
    sb.AppendFormat("Received image set {0}: ", model.Name);
    model.Images.ForEach(i =>
        sb.AppendFormat("Got image {0} of type {1} and size {2} bytes,", 
            i.FileName, 
            i.MimeType,
            i.ImageData.Length)
        );

    var result = sb.ToString();
    Trace.Write(result);

    return result;
}

Fortunately, WebAPI has a notion of MediaTypeFormatter, which basically lets you define logic for translating a certain type of request to a certain .NET type (and back). Here’s how to implement one for the ImageSet:

public class ImageSetMediaTypeFormatter : MediaTypeFormatter
{
    public ImageSetMediaTypeFormatter()
    {
        SupportedMediaTypes.Add(new MediaTypeHeaderValue("multipart/form-data"));
    }
        
    public override bool CanReadType(Type type)
    {
        return type == typeof (ImageSet);
    }

    public override bool CanWriteType(Type type)
    {
        return false;
    }

    public async override Task<object> ReadFromStreamAsync(
        Type type,
        Stream readStream, 
        HttpContent content, 
        IFormatterLogger formatterLogger)
    {
        var provider = await content.ReadAsMultipartAsync();

        var modelContent = provider.Contents
            .FirstOrDefault(c => c.Headers.ContentDisposition.Name.NormalizeName() == "imageset");
            
        var imageSet = await modelContent.ReadAsAsync<ImageSet>();

        var fileContents = provider.Contents
            .Where(c => c.Headers.ContentDisposition.Name.NormalizeName().Matches(@"image\d+"))
            .ToList();

        imageSet.Images = new List<Image>();
        foreach (var fileContent in fileContents)
        {
            imageSet.Images.Add(new Image
            {
                ImageData = await fileContent.ReadAsByteArrayAsync(),
                MimeType = fileContent.Headers.ContentType.MediaType,
                FileName = fileContent.Headers.ContentDisposition.FileName.NormalizeName()
            });
        }

        return imageSet;

    }
}

public static class StringExtenstions
{
    public static string NormalizeName(this string text)
    {
        return text.Replace("\"", "");
    }

    public static bool Matches(this string text, string pattern)
    {
        return Regex.IsMatch(text, pattern);
    }
}

By adding an entry to SupportedMediaTypes, you specify what content types this MediaTypeFormatter is able to handle. Then in CanRead and CanWrite you specify .NET types, to which (or from which) the request can be translated. ReadFromStreamAsync does the actual work.

Decoupling controller logic from request parsing logic gives you possibility to handle multiple request formats and you can let your clients choose the format they are most comfortable with by specifying appropriate Content-Type header.

Note: when you do the content.ReadAsMultipartAsync() call, you are using the MultipartMemoryStreamProvider, which handles all processing in-memory. Depending on your scenario, you may take different approach, for example MultipartFormDataStreamProvider, for which you’ll find a nice sample here.

The above code doesn’t do any request format validation for clarity of the example. In production, you’ll get all types of malformed requests from 3rd party clients, so you need to handle those situations.

How about some client code? First of all, on the client side we’ll make a small change in the ImageSet class:

    public class ImageSet
    {
        public string Name { get; set; }

        [JsonIgnore]
        public List<Image> Images { get; set; }
    }

We want the JSON serialization to ignore Images collection, since they will be put into separate request parts.

This is how you could prepare a request to be sent:

static void Main(string[] args)
{
    var imageSet = new ImageSet()
    {
        Name = "Model",
        Images = Directory
            .EnumerateFiles("../../../../../SampleImages")
            .Where(file => new[] {".jpg", ".png"}.Contains(Path.GetExtension(file)))
            .Select(file => new Image
            {
                FileName = Path.GetFileName(file),
                MimeType = MimeMapping.GetMimeMapping(file),
                ImageData = File.ReadAllBytes(file)
            })
            .ToList()
    };

    SendImageSet(imageSet);
}

And here’s how to send it using HttpClient and Json .NET:

private static void SendImageSet(ImageSet imageSet)
{
    var multipartContent = new MultipartFormDataContent();

    var imageSetJson = JsonConvert.SerializeObject(imageSet, 
        new JsonSerializerSettings
        {
            ContractResolver = new CamelCasePropertyNamesContractResolver()
        });

    multipartContent.Add(
        new StringContent(imageSetJson, Encoding.UTF8, "application/json"), 
        "imageset"
        );

    int counter = 0;
    foreach (var image in imageSet.Images)
    {
        var imageContent = new ByteArrayContent(image.ImageData);
        imageContent.Headers.ContentType = new MediaTypeHeaderValue(image.MimeType);
        multipartContent.Add(imageContent, "image" + counter++, image.FileName);
    }

    var response = new HttpClient()
        .PostAsync("http://localhost:53908/api/send", multipartContent)
        .Result;

    var responseContent = response.Content.ReadAsStringAsync().Result;
    Trace.Write(responseContent);
}

Summary

There are multiple ways to send mixed plain text / binary data to a REST API endpoint. The best what you can do while implementing public-facing API, is to let your clients choose format which is convenient for them. ASP.NET WebAPI has ways to facilitate that. multipart/form-data requests are a bit more complicated than whole-request binary serialization (BSON or protobuf), but may be more compatible with some platforms.

The code presented in this post can be found on Github. You’ll find there also client code samples for multipart/form-data scenario for Java, Python and node.js

 

[출처] http://blog.marcinbudny.com/2014/02/sending-binary-data-along-with-rest-api.html#.V6VG-mbr0y8

 

 

 

본 웹사이트는 광고를 포함하고 있습니다.
광고 클릭에서 발생하는 수익금은 모두 웹사이트 서버의 유지 및 관리, 그리고 기술 콘텐츠 향상을 위해 쓰여집니다.
번호 제목 글쓴이 날짜 조회 수
1195 [ 一日30分 인생승리의 학습법] VBA Web Scraping: How Can VBA Be Used To Scrape Website Data? file 졸리운_곰 2024.04.13 3
1194 [ 一日30分 인생승리의 학습법] 윈도우 실행파일 구조(PE파일) file 졸리운_곰 2024.03.31 3
1193 [ 一日30分 인생승리의 학습법] [Analysis] PE(Portable Executable) 파일 포맷 공부 file 졸리운_곰 2024.03.31 3
1192 [ 一日30分 인생승리의 학습법] 성공하는 메타버스의 3가지 조건 file 졸리운_곰 2024.03.30 7
1191 [ 一日30分 인생승리의 학습법] REST, REST API, RESTful 과 HATEOAS file 졸리운_곰 2024.03.10 9
1190 [ 一日30分 인생승리의 학습법] 렌더링 삼형제 CSR, SSR, SSG 이해하기 file 졸리운_곰 2024.03.10 2
1189 [ 一日30分 인생승리의 학습법] 엑셀 VBA에서 셀레니움 사용을 위한 Selenium Basic 설치 file 졸리운_곰 2024.02.23 11
1188 [ 一日30分 인생승리의 학습법]500 Lines or Less Blockcode: A Visual Programming Toolkit : 500줄 이하의 블록코드: 시각적 프로그래밍 툴킷 졸리운_곰 2024.02.12 4
1187 [ 一日30分 인생승리의 학습법] 구글 클라이언트(앱) 아이디를 발급받으려면 어떻게 해야 하나요? 졸리운_곰 2024.01.28 3
1186 [ 一日30分 인생승리의 학습법] 빅뱅 프로젝트를 성공적으로 오픈하기 위한 팁 졸리운_곰 2023.12.27 16
1185 [ 一日30分 인생승리의 학습법]“빅뱅 전환보다 단계적 전환 방식이 이상적 애자일팀과 협업 쉽게 체질 개선을” file 졸리운_곰 2023.12.27 12
1184 [ 一日30分 인생승리의 학습법] Big-bang / phased 접근 file 졸리운_곰 2023.12.27 3
1183 [ 一日30分 인생승리의 학습법] CodeDragon 메뉴 데이터 전환의 개념 이해 - 데이터 전환의 개념, 데이터 전환방식, 데이터 전환방식 및 장단점 비교, 데이터전환 이후 검토해야 할 사항 졸리운_곰 2023.12.27 5
1182 [ 一日30分 인생승리의 학습법] 블록체인과 IPFS를 이용한 안전한 데이터 공유 플랫폼 - 분쟁 해결 시스템 file 졸리운_곰 2023.12.27 6
1181 [ 一日30分 인생승리의 학습법] 블록체인과 IPFS를 이용한 안전한 데이터 공유 플랫폼 - 개념과 리뷰 시스템 file 졸리운_곰 2023.12.27 4
1180 [ 一日30分 인생승리의 학습법] 소켓 CLOSE_WAIT 발생 현상 및 처리 방안 file 졸리운_곰 2023.12.03 7
1179 [ 一日30分 인생승리의 학습법] robots 설정하기 졸리운_곰 2023.12.03 3
1178 [ 一日30分 인생승리의 학습법] A Tutorial and Elementary Trajectory Model for the Differential Steering System of Robot Wheel Actuators : 로봇 휠 액츄에이터의 차동 조향 시스템에 대한 튜토리얼 및 기본 궤적 모델 file 졸리운_곰 2023.11.29 6
1177 [ 一日30分 인생승리의 학습법] Streamline Your MLOps Journey with CodeProject.AI Server : CodeProject.AI 서버로 MLOps 여정을 간소화하세요 file 졸리운_곰 2023.11.25 2
1176 [ 一日30分 인생승리의 학습법] Comparing Self-Hosted AI Servers: A Guide for Developers / : 자체 호스팅 AI 서버 비교: 개발자를 위한 가이드 file 졸리운_곰 2023.11.25 10
대표 김성준 주소 : 경기 용인 분당수지 U타워 등록번호 : 142-07-27414
통신판매업 신고 : 제2012-용인수지-0185호 출판업 신고 : 수지구청 제 123호 개인정보보호최고책임자 : 김성준 sjkim70@stechstar.com
대표전화 : 010-4589-2193 [fax] 02-6280-1294 COPYRIGHT(C) stechstar.com ALL RIGHTS RESERVED