Hello, I'm running some kind of file upload server. One day I found that when I upload files using cURL in M1 MacBook Pro, the files are corrupted. Precisely, the file size is the same, but some parts of binary data are located in out of order.
At first, I thought that the server codes have a bug. I dug in it for hours and hours, but I cannot find any points. So I decided to write the very simple server in Golang and Python which just get a file from multipart request and test corruption.
Golang:
package main
import (
"io"
"io/ioutil"
"log"
"mime/multipart"
"net/http"
"os"
"github.com/gorilla/mux"
)
func main() {
muxer := mux.NewRouter()
muxer.HandleFunc("/upload", uploadHandler).Methods("POST")
server := &http.Server{
Addr: "0.0.0.0:8888",
Handler: muxer,
}
log.Printf("start server")
if err := server.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("failed to start server: %s", err.Error())
}
}
func uploadHandler(w http.ResponseWriter, r *http.Request) {
var reader *multipart.Reader
var part *multipart.Part
var err error
if reader, err = r.MultipartReader(); err != nil {
log.Fatalf("failed to get multipart reader")
}
for {
if part, err = reader.NextPart(); err != nil {
if err == io.EOF {
break
} else {
log.Fatalf("failed to get next part")
}
}
if part.FormName() == "file" {
readAndTest(part)
}
if err = part.Close(); err != nil {
log.Fatalf("failed to close part")
}
}
}
func readAndTest(part *multipart.Part) {
log.Printf("read file")
buf, err := ioutil.ReadAll(part)
if err != nil {
log.Fatalf("failed to read file")
}
f, err := os.Open("60.m4a")
if err != nil {
log.Fatalf("failed to open target file")
}
defer f.Close()
tgt, err := ioutil.ReadAll(f)
if err != nil {
log.Fatalf("failed to read target file")
}
if len(buf) != len(tgt) {
log.Fatalf("length is not the same")
}
for i := 0; i < len(buf); i++ {
if buf[i] != tgt[i] {
log.Fatalf("data[%d] is not the same", i)
}
}
log.Printf("file is read successfully")
}
Python:
from flask import Flask, render_template, request
import logging
app = Flask(__name__)
@app.route('/upload', methods = ['POST'])
def upload_file():
if request.method == 'POST':
logging.debug("read file")
f = request.files['file']
buf = f.read()
with open("60.m4a", "rb") as f_tgt:
tgt = f_tgt.read()
if len(buf) != len(tgt):
raise RuntimeError("length is not the same")
for i in range(len(buf)):
if buf[i] != tgt[i]:
raise RuntimeError("data[%d] is not the same" %i)
logging.debug("file is read successfully")
return 'file uploaded successfully'
if __name__ == '__main__':
app.run(host="0.0.0.0", port="8888")
(60.m4a
is a 59MB audio file that I tested)
I ran them in the remote CentOS 7 server and sent upload request using cURL in my M1 MacBook Pro like below:
curl -X POST "http://domain_of_server:8888/upload" -F "file=@60.m4a"
The problem was occurred in both of Golang and Python server. It doesn't happen every time, but quite often in my experience.
I'm not sure that I used M1 is the main reason, but when I tried the same cURL request in my Intel MacBook Pro with the same network environment, the same cURL version (7.64.1) and the same OS version (Big Sur 11.4), it never happened till now.
I cannot decide which makes this problem in my M1 MacBook Pro. Are there anybody who suffers similar problem?