MemoryError with LBT Radiance Recipes

OlivierDambron · May 4, 2021, 10:37pm

Thanks for these tips, they make a lot of sense.

I’m testing a Direct sun hours case with a 332mb hbpkl, 493916 points, sensor count at 16463, using 30 workers (my hardware has 20 cpus with double thread, and am just putting to work 30 threads, and 64 GB).
It is returning a memory error of which I attach the log file here logs.log (55.9 KB)

Would you know what would be the bottleneck here?

OlivierDambron · May 4, 2021, 11:10pm

I found that even with 1 point in my grid, I was getting a memory issue.

It seemed that issue was within the hbjson or hbpkl.

I was initially meshing my breps into the hb_face component. However it detects an error as follows :
“1. Solution exception:There must be at least 3 vertices for a Face3D boundary. Got 2”

It seems that I will need to find a way to narrow down this error in the large geometry

chris · May 4, 2021, 11:45pm

I know this isn’t the most helpful answer but it looks pretty clear that you are just running out of memory when trying to load the Model sensor grids from the .hbpkl file. Are you able to re-load the model into Grasshopper from the .hbpkl file? That will help us see if it’s something specific to how cPython is loading things vs. IronPython.

OlivierDambron · June 1, 2021, 4:46pm

hi @chris

The solution to the previous error I detected was to adjust the tolerance and the units.
This occurs when doubleclicking on a GH file directly , as it opens a default Rhino file (ie: Large Objects, meters vs. Small Objects, meters) that may differ from one computer to another.

While working on these large models (213 933 surfaces, 9 250 000 sensors), this time I encountered a limitation prior to being able to write the hbjson file. All components upstream have worked until this one.

Would you know what causes this memory exception?

Best,
O

chris · June 1, 2021, 6:22pm

Hey @OlivierDambron ,

That’s the result of the fact that we output a JSON file that keeps track of all the sensor grids in the model:

github.com

ladybug-tools/honeybee-radiance/blob/master/honeybee_radiance/writer.py#L380-L402


def _write_sensor_grids(folder, model, grids_filter):
    """Write out the files that need to go into any dynamic model folder.

    Args:
        folder: The sensor grids folder.
        model: A Honeybee model.
        grids_filter: A list of sensor grid names to filter the sensor grids in the
            model. Use this argument to indicate specific sensor grids that should
            be included. By default all the sensor grids will be exported. You can use
            wildcard symbols in names. Use relative path from inside grids folder.

    Returns:
        A tuple for path to _info.json and _model_grids_info.json. The first file
        are the information for the sensor grids that are written to the folder and
        the second one is the information for the input sensor grids from the model.

        Use ``_info.json`` for access the sensor grid information for running the
        commands and ``_model_grids_info`` file for loading the results back to match
        with the model. Model_grids_info has an extra key for `start_ln` which provides
        the start line for where the sensors for this grid starts in a pts file. Unless

This file has been truncated. show original

This file gets used by all of the grid-based recipes to help the process understand all of the grids to be simulated.

So the short answer is that your model has too many sensor grids and you might consider joining some together. I don’t know if @mostapha has any thoughts on making this grid_info.json file optional in the output rad folder but removing it probably means that the resulting rad folder cannot work with any of the current recipes.

OlivierDambron · June 2, 2021, 10:50am

hi @chris

I’m afraid I’ve been flattening all points generated to create a single grid, that way I can reduce the overhead time of decomposing and recomposing subgrids as per your explanation here : Rules for optimal sensor count setting with lbt-recipes - #3 by chris

Yet the error persists.

chris · June 2, 2021, 5:08pm

I think my best suggestion for a quick fix right now is just to split up your sensor grid into multiple simulations. For a longer term fix, I don’t know if we’ll really be able to find any satisfactory solution since you are really pushing the limits of file size but maybe we find something that works if you post a link to a sample that lets us recreate the issue.

OlivierDambron · June 4, 2021, 3:26pm

hi @chris

Thanks, here is a sample file. https://we.tl/t-8lVFts7dVY
(To avoid wetransfer links when uploading larger files, would you have a recommendation?)

It is quite big, the context geometry is plugged in, however I left the test surfaces unplugged to avoid waiting too much.

After writing the case I would navigate into the folder via the command line and run:
“lbt-recipes run direct_sun_hours inputs.json --workers 7”

chris · August 11, 2021, 11:36pm

2 posts were split to a new topic: Memory Error when Creating Rooms