From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM autolearn=unavailable autolearn_force=no version=3.4.4 X-Received: by 2002:a37:8c84:: with SMTP id o126mr43541785qkd.467.1568345332611; Thu, 12 Sep 2019 20:28:52 -0700 (PDT) X-Received: by 2002:aca:90b:: with SMTP id 11mr1522734oij.37.1568345331921; Thu, 12 Sep 2019 20:28:51 -0700 (PDT) Path: eternal-september.org!reader01.eternal-september.org!feeder.eternal-september.org!weretis.net!feeder6.news.weretis.net!feeder.usenetexpress.com!feeder-in1.iad1.usenetexpress.com!border1.nntp.dca1.giganews.com!nntp.giganews.com!o24no1622066qtl.0!news-out.google.com!q23ni67qtl.1!nntp.google.com!o24no1622062qtl.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail Newsgroups: comp.lang.ada Date: Thu, 12 Sep 2019 20:28:51 -0700 (PDT) Complaints-To: groups-abuse@google.com Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=2601:193:4103:71a0:6cd1:5efd:23f:7ad5; posting-account=1tLBmgoAAAAfy5sC3GUezzrpVNronPA- NNTP-Posting-Host: 2601:193:4103:71a0:6cd1:5efd:23f:7ad5 User-Agent: G2/1.0 MIME-Version: 1.0 Message-ID: <96ceadae-90dd-4781-9223-3a50ad1668ef@googlegroups.com> Subject: libcurl with Ada - how to HTTP POST transfer with large file upload From: Matt Borchers Injection-Date: Fri, 13 Sep 2019 03:28:52 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Xref: reader01.eternal-september.org comp.lang.ada:57142 Date: 2019-09-12T20:28:51-07:00 List-Id: A recent situation has caused some previously working code (written in Ada = 95 era) to stop working. What has changed is updating from using a very ol= d compiler (GNAT 6.3.0) and AWS (Ada Web Server) v2.3.0 to the latest avail= able. We are using libcurl v7.19.6 (quite old) to transfer files using unauthenti= cated HTTP. I have tried to build with the latest available libcurl librar= y files. Although the executable builds, the executable will not run. So = for now, at least, I am stuck with the v7.19.6. We are also using the AdaCurl binding by Andreas Almroth that is dated 2005= -05-24. I have found a few errors with this binding recently in my investi= gation of how to fix my problem. The files we transfer can be small to very large. The old mechanism that w= as in place was this: We set up a libcurl EASY session with PUT. If the f= ile size was less than 512MB then a buffer was created in memory and that b= uffer written to disk. If the file was larger then we read the message bod= y directly from the socket and wrote that data to disk. The message body can no longer be read directly from the socket when using = the new AWS. I am told that in order to avoid a memory buffer that I shoul= d send the file as an attachment to a POST request. My goal is to transfer any file with libcurl to AWS in the fastest way poss= ible while avoiding a temporary memory buffer. I have tried many things over the past week and have had no luck in getting= libcurl to perform how I want it to so that AWS can retrieve the data from= the message body. The libcurl documentation and its debug routine option = is lackluster in that it does not go in-depth enough to help people underst= and how to correct what is wrong. I have tried coding up the many examples= available but the examples don't work. I undertand that when it is workin= g properly the file will be written directly to disk and the HTTP response = handler will be provided with the filename as a FORM parameter. Does anybody have a working example of using libcurl with or without AWS as= I described? Perhaps a better alternative to libcurl for my situation? B= elow is the latest code I have which I've read is on the right track. I do= n't really understand the adacurl.multi package which is used below with th= e adacurl.easy package. Thank you for any assistance. Matt declare hndl : CURL_P; mhndl : CURLM_P; mc : CURLMcode; still_run : aliased INTERFACES.C.INT :=3D 0; u32_str : STRING :=3D UNSIGNED_32'Image( u32 ); ca : aliased CHAR_ARRAY :=3D to_c( "http://.../database/" & u32_= str(2..u32_str'Last) ); continue : aliased CHAR_ARRAY :=3D to_c( "Expect:" ); rcode : aliased LONG :=3D 0; s500 : constant :=3D 500; form, ftail : CURL_HTTPPOST_P; fcode : CURLFORMcode; dbid : aliased CHAR_ARRAY :=3D to_c( u32_str(2..u32_str'Last) ); sendfile : aliased CHAR_ARRAY :=3D to_c( "sendfile" ); filename : aliased CHAR_ARRAY :=3D to_c( "filename" ); submit : aliased CHAR_ARRAY :=3D to_c( "submit" ); send : aliased CHAR_ARRAY :=3D to_c( "send" ); begin hndl :=3D curl_easy_init; if hndl =3D NULL then text_io.put_line( "Curl failed to initialize" ); return; end if; mhndl :=3D curl_multi_init; if mhndl =3D NULL then text_io.put_line( "Curl-Multi failed to initialize" ); return; end if; code :=3D curl_easy_setopt( hndl, CURLOPT_VERBOSE, LONG(1) ); code :=3D curl_easy_setopt( hndl, CURLOPT_DEBUGFUNCTION, curl_debug'Unr= estricted_Access ); code :=3D curl_easy_setopt( hndl, CURLOPT_ERRORBUFFER, to_chars_ptr( er= rbuf'Unrestricted_Access ) ); code :=3D curl_easy_setopt( hndl, CURLOPT_READDATA, fd'Address ); code :=3D curl_easy_setopt( hndl, CURLOPT_READFUNCTION, wrap_fread'Unre= stricted_Access ); code :=3D curl_easy_setopt( hndl, CURLOPT_WRITEFUNCTION, ignore_respons= e'Unrestricted_Access ); code :=3D curl_easy_setopt( hndl, CURLOPT_POST, LONG(1) ); code :=3D curl_easy_setopt( hndl, CURLOPT_URL, to_chars_ptr( ca'Unrestr= icted_Access ) ); if code /=3D CURLE_OK then text_io.put_line( "Failed to set url, code is " & CURLCODE'Image( c= ode ) ); return; end if; headers :=3D curl_slist_append( headers, to_chars_ptr( continue'Unrestr= icted_Access ) ); code :=3D curl_easy_setopt( hndl, CURLOPT_HTTPHEADER, headers.all'Ad= dress ); fcode :=3D curl_formadd( form'Unrestricted_Access, ftail'Unrestricted_A= ccess, CURLFORM_COPYNAME, to_chars_ptr( sendfile'Unrestricted_Access ), CURLFORM_FILE, to_chars_ptr( dbid'Unrestricted_Access ), CURLFORM_END ); fcode :=3D curl_formadd( form'Unrestricted_Access, ftail'Unrestricted_A= ccess, CURLFORM_COPYNAME, to_chars_ptr( filename'Unrestricted_Access ), CURLFORM_COPYCONTENTS, to_chars_ptr( dbid'Unrestricted_Access ), CURLFORM_END ); fcode :=3D curl_formadd( form'Unrestricted_Access, ftail'Unrestricted_A= ccess, CURLFORM_COPYNAME, to_chars_ptr( submit'Unrestricted_Access ), CURLFORM_COPYCONTENTS, to_chars_ptr( send'Unrestricted_Access ), CURLFORM_END ); code :=3D curl_easy_setopt( hndl, CURLOPT_HTTPPOST, form.all'Address ); mc :=3D curl_multi_add_handle( mhndl, hndl ); mc :=3D curl_multi_perform( mhndl, still_run'Unrestricted_Access ); while still_run /=3D 0 loop declare fdread, fdwrite, fdexcep : aliased FD_SET :=3D (others =3D> 0); maxfd : aliased INTERFACES.C.INT :=3D -1; begin mc :=3D curl_multi_fdset( mhndl, fdread'Unrestricted_Access, fdwrite'Unrestricted_Access, fdexcep'Unrestricted_Access, maxfd'Unrestricted_Access ); if mc /=3D CURLM_OK then text_io.put_line( "curl_multi_fdset() failed, code =3D" & m= c'imaage ); exit; end if; delay 0.1; mc :=3D curl_multi_perform( mhndl, still_run'Unrestricted_Acces= s ); if mc =3D CURLM_OK then code :=3D CURLE_OK; else code :=3D CURL_LAST; end if; end; end loop; if code =3D CURLE_OK then code :=3D curl_easy_getinfo( hndl, CURLINFO_RESPONSE_CODE, rcode'Ac= cess ); if code =3D CURLE_OK and then rcode >=3D s500 then text_io.put_line( "Server error executing curl for PUT_FILE res= ponse code :" & rcode'Image ); code :=3D CURLE_SEND_ERROR; end if; end if; mc :=3D curl_multi_cleanup( mhndl ); curl_formfree( form ); curl_slist_free_all( headers ); curl_easy_cleanup( hndl ); end;