problems parse (large) json file with gnatcoll.json

problems parse (large) json file with gnatcoll.json

[Björn Lundin]

Hi!
I got a json file which is sized around 4 Mb

$ ls -la menu.json
-rwxr--r-- 1 bnl bnl 4814417 apr 18 17:07 menu.json

Platform is Linux 64-bit (debian)

My problem is that I don't see a way to open
a file and parse it, I just see the possibility to parse a string.

And reading the file into a string gives Storage_Error.
Or actually, declaring a string sub_type does.


function Load_File(Filename : in String) return String is
   use Ada.Directories;
   File_Size    : constant Natural := Natural (Size (Filename));
   subtype Test_JSON_Str is String (1 .. File_Size);   <-- Storage_Error



ulimit is at 8192 kb and I increase it to 16384
(that is max what I can see)
$ ulimit -s
16384

So - what is my options here?

(The json file and its size is out of my control)

--
Björn

Re: problems parse (large) json file with gnatcoll.json

[Jeffrey R. Carter]

On 04/18/2016 11:21 AM, Björn Lundin wrote:
>
> function Load_File(Filename : in String) return String is
>    use Ada.Directories;
>    File_Size    : constant Natural := Natural (Size (Filename));
>    subtype Test_JSON_Str is String (1 .. File_Size);   <-- Storage_Error

I doubt the subtype declaration gives you Storage_Error. It should be if you 
declare an object of the subtype:

Content : Test_JSON_Str;

You can probably get around this by declaring an access type:

type Str_Ptr is access Test_JSON_Str;

Content : constant Str_Ptr := new Test_JSON_Str;

-- 
Jeff Carter
"We call your door-opening request a silly thing."
Monty Python & the Holy Grail
17

Re: problems parse (large) json file with gnatcoll.json

[joakimds]

Den måndag 18 april 2016 kl. 20:29:07 UTC+2 skrev björn lundin:
> Hi!
> I got a json file which is sized around 4 Mb
> 
> $ ls -la menu.json
> -rwxr--r-- 1 bnl bnl 4814417 apr 18 17:07 menu.json
> 
> Platform is Linux 64-bit (debian)
> 
> My problem is that I don't see a way to open
> a file and parse it, I just see the possibility to parse a string.
> 
> And reading the file into a string gives Storage_Error.
> Or actually, declaring a string sub_type does.
> 
> 
> function Load_File(Filename : in String) return String is
>    use Ada.Directories;
>    File_Size    : constant Natural := Natural (Size (Filename));
>    subtype Test_JSON_Str is String (1 .. File_Size);   <-- Storage_Error
> 
> 
> 
> ulimit is at 8192 kb and I increase it to 16384
> (that is max what I can see)
> $ ulimit -s
> 16384
> 
> So - what is my options here?
> 
> (The json file and its size is out of my control)
> 
> --
> Björn

Dear Björn,

Not sure I have fully understood the problem, but if the problem is somehow connected with the internal implementation of GNATColl's json parsing, maybe Tero Koskinens Json library does not suffer from the same issue:

https://bitbucket.org/tkoskine/jdaughter

Best regards,
Joakim

Re: problems parse (large) json file with gnatcoll.json

[Björn Lundin]

On 2016-04-18 20:51, Jeffrey R. Carter wrote:
> I doubt the subtype declaration gives you Storage_Error. It should be if
> you declare an object of the subtype:

Of course you are correct.
I wonder how I misread the stacktrace that much.
Anyway your tip paid off.

> You can probably get around this by declaring an access type:


The loading function now looks like

  function Load_File(Filename : in String) return String is
     use Ada.Directories;
     File_Size    : constant Natural := Natural (Size (Filename));
     subtype JSON_String is String (1 .. File_Size);
     type String_Ptr is access JSON_String;
     Content : constant String_Ptr := new JSON_String;
     package File_IO is new Ada.Direct_IO(JSON_String);
     File : File_IO.File_Type;
  begin
     File_IO.Open (File => File, Mode => File_IO.In_File,
                   Name => Filename);
     File_IO.Read (File => File, Item => Content.all);
     File_IO.Close(File => File);
     return Content.all;
  end Load_File;
  --------------------------

and it works. Thanks. (I'll fix the leak later on)

The Load_File function is from

http://wiki.ada-dk.org/index.php/Handling_JSON_Using_GNATColl


--
Björn

Re: problems parse (large) json file with gnatcoll.json

[Björn Lundin]

On 2016-04-18 20:59, joakimds@kth.se wrote:

> Not sure I have fully understood the problem, but if the problem is somehow connected with the internal implementation of GNATColl's json parsing, maybe Tero Koskinens Json library does not suffer from the same issue:
> 
> https://bitbucket.org/tkoskine/jdaughter

Thanks but Jeffrey Carter solved it for me.


-- 
--
Björn

Re: problems parse (large) json file with gnatcoll.json

[Jacob Sparre Andersen]

Björn Lundin <b.f.lundin@gmail.com> writes:

> ulimit is at 8192 kb and I increase it to 16384
> (that is max what I can see)
> $ ulimit -s
> 16384

Have you tried to increase it a bit more?  I have occasionally set the
stack size to 4 Gb for some of my Ada programs.

I think GNAT allows you to request a specific stack size with a pragma,
but I can't remember the details.

Greetings,

Jacob
-- 
"War does not determine who is right - only who is left."
                                         -- Bertrand Russell

Re: problems parse (large) json file with gnatcoll.json

[Björn Lundin]

On 2016-04-19 10:17, Jacob Sparre Andersen wrote:
> Björn Lundin <b.f.lundin@gmail.com> writes:
> 
>> ulimit is at 8192 kb and I increase it to 16384
>> (that is max what I can see)
>> $ ulimit -s
>> 16384
> 
> Have you tried to increase it a bit more?  I have occasionally set the
> stack size to 4 Gb for some of my Ada programs.

How?
I got errors when setting it higher (at prompt level)


> I think GNAT allows you to request a specific stack size with a pragma,
> but I can't remember the details.

Yes, at least for tasks.

    task Sender_Task is
      pragma Storage_Size(600 * 1024);
      entry Start;
      entry Stop;
    end Sender_Task;


Perhaps for the environment (main) task as well.

However, Jeff solved my actual problem.

-- 
--
Björn

Re: problems parse (large) json file with gnatcoll.json

[gautier_niouzes]

Alternatively to the access type option, you can use an Unbounded_String; then the heap allocation *and* deallocation is done for you behind the scenes.
_________________________ 
Gautier's Ada programming 
http://gautiersblog.blogspot.com/search/label/Ada 
NB: follow the above link for a valid e-mail address 

Re: problems parse (large) json file with gnatcoll.json

[G.B.]

On 19.04.16 11:56, gautier_niouzes@hotmail.com wrote:
> Alternatively to the access type option, you can use an Unbounded_String; then the heap allocation *and* deallocation is done for you behind the scenes.

It was sad, though, when the makers of GCC/GNAT had
started to make Ada programs subject to unforeseeable stack
size implications that would make pretty normal programs
raise Storage_Error in an unpredictable way. As is frequently
the case, "good reasons" were stated for giving up plain
old arrays in favor of explicit memory management, or for
suggesting the use of more complex data structures; IIRC,
this has to do with implementation decisions in the C
compiler market in order to attack problems at that level
of implementation.

For example, the function below uses an array of
dynamic size, and will raise at some point depending
on the value of Size at run-time:

    function Using_Local_String (Size : Positive) return Boolean
    is
       subtype S is String (1 .. Size);
       X : S;
    begin
       X (X'First) := Character'Max('A', X (X'Last));
       return X (X'First) = 'A';
    end Using_Local_String;

Try approximating the "Bang!" size:

    for N in 1 .. 24 loop
       X := Big_String.Using_Local_String (Size => 2**N);
       Text_Io.Put_Line
         (Item => Positive'Image(2**N) & " -> " & Boolean'Image (X));
    end loop;

Mine is here:
   4194304 -> TRUE

raised STORAGE_ERROR : stack overflow

Re: problems parse (large) json file with gnatcoll.json

[Shark8]

On Monday, April 18, 2016 at 1:12:49 PM UTC-6, björn lundin wrote:
> 
> The loading function now looks like
> 
>   function Load_File(Filename : in String) return String is
>      use Ada.Directories;
>      File_Size    : constant Natural := Natural (Size (Filename));
>      subtype JSON_String is String (1 .. File_Size);
>      type String_Ptr is access JSON_String;
>      Content : constant String_Ptr := new JSON_String;
>      package File_IO is new Ada.Direct_IO(JSON_String);
>      File : File_IO.File_Type;
>   begin
>      File_IO.Open (File => File, Mode => File_IO.In_File,
>                    Name => Filename);
>      File_IO.Read (File => File, Item => Content.all);
>      File_IO.Close(File => File);
>      return Content.all;
>   end Load_File;
>   --------------------------

Just replace the return w/ an extended return containing an instance of Unchecked_Deallocation; eg:

return Result : JSON_String := Content.all do
  Free( Content ); -- "Free" is the instantiation of Unchecked_Deallocation.
end return;

*BUT*

You really shouldn't need it, IIRC when the type String_Ptr goes out of scope all instances should become invalid, thus allowing the compiler to reclaim it's memory from the storage pool.

Re: problems parse (large) json file with gnatcoll.json

[AdaMagica]

Am Dienstag, 19. April 2016 21:48:35 UTC+2 schrieb Shark8:
> return Result : JSON_String := Content.all do
>   Free( Content ); -- "Free" is the instantiation of Unchecked_Deallocation.
> end return;
> 
> *BUT*
> 
> You really shouldn't need it, IIRC when the type String_Ptr goes out of scope all instances should become invalid, thus allowing the compiler to reclaim it's memory from the storage pool.

Attention! When an access type goes out of scope, all target objects are finalized, but the storage is only reclaimed if there is a size clause on the access type.

Re: problems parse (large) json file with gnatcoll.json

[Björn Lundin]

On 2016-04-19 21:48, Shark8 wrote:
> 
> return Result : JSON_String := Content.all do
>   Free( Content ); -- "Free" is the instantiation of Unchecked_Deallocation.
> end return;
> 
> *BUT*
> 
> You really shouldn't need it, IIRC when the type String_Ptr goes out of scope all instances should become invalid, thus allowing the compiler to reclaim it's memory from the storage pool.
> 

That implies that there is garbage collection in the compiler.
Is that really something I can trust is correct ?


Anyway, the final solution (or last incarnation of it)
is not reading from file, as the problem originates from.
Turns out that the total solutions becomes easier if
read directly from the web



 AWS_Reply := Aws.Client.Get (Url          => Token.URL_NAVIGATION_DATA,
                              Headers      => HTTP_Headers,
                              Timeouts     =>
                               Aws.Client.Timeouts (Each => 1200.0));

 Menu := Read (Strm     => Aws.Response.Message_Body(AWS_Reply),
               Filename => "");



-- 
--
Björn

Re: problems parse (large) json file with gnatcoll.json

[briot.emmanuel]

On Monday, April 18, 2016 at 9:12:49 PM UTC+2, björn lundin wrote:
> The loading function now looks like
> 
>   function Load_File(Filename : in String) return String is

Since you are using GNATCOLL, you might as well go all the way and look at the
other packages there. In particular, GNATCOLL.Mmap or GNATCOLL.VFS.Read_File
provide the ability to efficiently read a file from the disk

Re: problems parse (large) json file with gnatcoll.json

[Björn Lundin]

On 2016-04-25 12:38, briot.emmanuel@gmail.com wrote:
> On Monday, April 18, 2016 at 9:12:49 PM UTC+2, björn lundin wrote:
>> The loading function now looks like
>>
>>   function Load_File(Filename : in String) return String is
> 
> Since you are using GNATCOLL, you might as well go all the way and look at the
> other packages there. In particular, GNATCOLL.Mmap or GNATCOLL.VFS.Read_File
> provide the ability to efficiently read a file from the disk
> 

Thanks Emmanuel.
Not for this task,
but the mmap looks interesting for other.

-- 
--
Björn