Anybody suffers uploaded files using cURL are corrupted in M1 MacBook Pro?

Hello, I'm running some kind of file upload server. One day I found that when I upload files using cURL in M1 MacBook Pro, the files are corrupted. Precisely, the file size is the same, but some parts of binary data are located in out of order.

At first, I thought that the server codes have a bug. I dug in it for hours and hours, but I cannot find any points. So I decided to write the very simple server in Golang and Python which just get a file from multipart request and test corruption.

Golang:

package main

import (
    "io"
    "io/ioutil"
    "log"
    "mime/multipart"
    "net/http"
    "os"

    "github.com/gorilla/mux"
)

func main() {
    muxer := mux.NewRouter()
    muxer.HandleFunc("/upload", uploadHandler).Methods("POST")

    server := &http.Server{
        Addr:    "0.0.0.0:8888",
        Handler: muxer,
    }

    log.Printf("start server")

    if err := server.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatalf("failed to start server: %s", err.Error())
    }
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
    var reader *multipart.Reader
    var part *multipart.Part
    var err error

    if reader, err = r.MultipartReader(); err != nil {
        log.Fatalf("failed to get multipart reader")
    }

    for {
        if part, err = reader.NextPart(); err != nil {
            if err == io.EOF {
                break
            } else {
                log.Fatalf("failed to get next part")
            }
        }

        if part.FormName() == "file" {
            readAndTest(part)
        }

        if err = part.Close(); err != nil {
            log.Fatalf("failed to close part")
        }
    }
}

func readAndTest(part *multipart.Part) {
    log.Printf("read file")

    buf, err := ioutil.ReadAll(part)
    if err != nil {
        log.Fatalf("failed to read file")
    }

    f, err := os.Open("60.m4a")
    if err != nil {
        log.Fatalf("failed to open target file")
    }
    defer f.Close()

    tgt, err := ioutil.ReadAll(f)
    if err != nil {
        log.Fatalf("failed to read target file")
    }

    if len(buf) != len(tgt) {
        log.Fatalf("length is not the same")
    }

    for i := 0; i < len(buf); i++ {
        if buf[i] != tgt[i] {
            log.Fatalf("data[%d] is not the same", i)
        }
    }

    log.Printf("file is read successfully")
}

Python:

from flask import Flask, render_template, request
import logging

app = Flask(__name__)

@app.route('/upload', methods = ['POST'])
def upload_file():
    if request.method == 'POST':
        logging.debug("read file")

        f = request.files['file']
        buf = f.read()

        with open("60.m4a", "rb") as f_tgt:
            tgt = f_tgt.read()

        if len(buf) != len(tgt):
            raise RuntimeError("length is not the same")

        for i in range(len(buf)):
            if buf[i] != tgt[i]:
                raise RuntimeError("data[%d] is not the same" %i)

        logging.debug("file is read successfully")

        return 'file uploaded successfully'

if __name__ == '__main__':
    app.run(host="0.0.0.0", port="8888")

(60.m4a is a 59MB audio file that I tested)

I ran them in the remote CentOS 7 server and sent upload request using cURL in my M1 MacBook Pro like below:

curl -X POST "http://domain_of_server:8888/upload" -F "file=@60.m4a"

The problem was occurred in both of Golang and Python server. It doesn't happen every time, but quite often in my experience.

I'm not sure that I used M1 is the main reason, but when I tried the same cURL request in my Intel MacBook Pro with the same network environment, the same cURL version (7.64.1) and the same OS version (Big Sur 11.4), it never happened till now.

I cannot decide which makes this problem in my M1 MacBook Pro. Are there anybody who suffers similar problem?

Are you using the curl tool that’s built in to macOS? Or a custom one, perhaps installed by Homebrew?

If it’s the former, I recommend that you use a packet trace to confirm that curl is sending the data in the wrong order. Once you’ve done that, you should file a bug against macOS’s curl tool, making sure to attach your packet trace.

Please post your bug number, just for the record.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Anybody suffers uploaded files using cURL are corrupted in M1 MacBook Pro?
 
 
Q