The dependency acquisition process of URBANopt is extremely slow during its first use

ZhengrongTao · May 13, 2025, 4:26pm

hi all.

When Dragonfly conducts its first energy consumption calculation using Urbanopt, it needs to be connected to the network. After the connection is established, it will retrieve the dependencies from http://rubygems.org/.

In the vast majority of areas, this process can be completed in just a few minutes. However, in areas with network restrictions, it might not be possible to directly obtain the required dependencies from this site.

This is the report related to the failure of dependency pull.

Defaulting to .bundle/install (and ignoring system wide: Bundler.configured_bundle_path.base_path=C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0)
Bundle final path is set to: D:/Desktop/test sample/.bundle/install
Defaulting to .bundle/install (and ignoring system wide: Bundler.configured_bundle_path.base_path=C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0)
Using runner options from runner.conf file
Merging in passed-in options
Initializing runner with dirname: 'D:/Desktop/test sample' and options: {:file_version=>"0.1.0", :max_datapoints=>1000000000, :num_parallel=>16, :run_simulations=>true, :verbose=>false, :gemfile_path=>"", :bundle_install_path=>"D:/Desktop/test sample/.bundle/install"}
@bundle_without_string = ''
needs_config = true
needs_platform = true
Error running command: 'bundle lock --add_platform ruby'
stdout: Fetching source index from http://rubygems.org/

stderr: Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from http://rubygems.org/ due to underlying error <IO::TimeoutError: Failed to open TCP connection to rubygems.org:80 (Blocking operation timed out!) (http://rubygems.org/specs.4.8.gz)>
Network error while fetching
http://rubygems.org/quick/Marshal.4.8/addressable-0.1.0.gemspec.rz (Failed to
open TCP connection to rubygems.org:80 (execution expired))
needs_update = true

When after a long period of time the dependencies still cannot be fully obtained, the program will stop and report the following content.

The URBANopt simulation failed to run.
No results were found at:
C:\Users\Tao\simulation\Buffalo_New_Development\run\honeybee_scenario
Top level ::CompositeIO is deprecated, require 'multipart/post' and use `Multipart::Post::CompositeReadIO` instead!
Top level ::Parts is deprecated, require 'multipart/post' and use `Multipart::Post::Parts` instead!
<internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- openstudio/common_measures (LoadError)
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/Users/Tao/simulation/Buffalo_New_Development/mappers/Honeybee.rb:33:in `<top (required)>'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:93:in `block in load_mapper_files'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:92:in `each'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:92:in `load_mapper_files'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:36:in `initialize'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:497:in `new'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:497:in `run_func'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:1270:in `<module:CLI>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:23:in `<module:URBANopt>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:22:in `<top (required)>'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/bin/uo:3:in `<top (required)>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/bin/uo:25:in `load'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/bin/uo:25:in `<main>'

and.

Runtime error (PythonException): The URBANopt simulation failed to run.
No results were found at:
C:\Users\Tao\simulation\Buffalo_New_Development\run\honeybee_scenario
Top level ::CompositeIO is deprecated, require 'multipart/post' and use `Multipart::Post::CompositeReadIO` instead!
Top level ::Parts is deprecated, require 'multipart/post' and use `Multipart::Post::Parts` instead!
<internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- openstudio/common_measures (LoadError)
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/Users/Tao/simulation/Buffalo_New_Development/mappers/Honeybee.rb:33:in `<top (required)>'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:93:in `block in load_mapper_files'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:92:in `each'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:92:in `load_mapper_files'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-scenario-1.0.0/lib/urbanopt/scenario/scenario_csv.rb:36:in `initialize'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:497:in `new'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:497:in `run_func'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:1270:in `<module:CLI>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:23:in `<module:URBANopt>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/lib/uo_cli.rb:22:in `<top (required)>'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:C:/URBANopt-cli-1.0.1/ruby/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/gems/urbanopt-cli-1.0.1/bin/uo:3:in `<top (required)>'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/bin/uo:25:in `load'
	from C:/URBANopt-cli-1.0.1/gems/ruby/3.2.0/bin/uo:25:in `<main>'


Traceback:
  line 918, in _output_urbanopt_files, "C:\Program Files\ladybug_tools\python\Lib\site-packages\dragonfly_energy\run.py"
  line 342, in run_urbanopt, "C:\Program Files\ladybug_tools\python\Lib\site-packages\dragonfly_energy\run.py"
  line 153, in script

Also, if the initial dependency pull fails and the contents in the C:\Users\[Username]\.bundle folder are not cleaned up, when re-executing URBANopt, it will not re-pull and verify again in the new case, but will directly report.

I would like to know if there is any way to switch to pulling dependencies from the mirror source instead of using http://rubygems.org/. Or is it any method that allows these dependencies to be manually added to URBANopt instead of being downloaded online?

best.
Zhengrong

MingboPeng · May 13, 2025, 4:40pm

Can you try to run this in your URBANopt gem folder?
gem sources --add https://your-mirror-server.com

ZhengrongTao · May 13, 2025, 4:57pm

hi @MingboPeng

I failed. This is unlike Python. It is not very convenient to add this command in a pre-defined environment. I attempted to use CMD in this folder to replace the required image source, but I failed.

chris · May 13, 2025, 6:19pm

Thanks for letting us know, @ZhengrongTao .

The download of the ~0.5 GB of Ruby gems for every URBANopt project folder is something that I know needs to eventually be addressed and I am waiting to get NREL’s thoughts on two possible solutions that I suggested. But both of them would take some time to implement so they aren’t going to be of immediate help.

In the meantime, I sent some of the NREL developers a link to this discussion in case they have any suggestions for you to try. I am sure that bundle has some ways to bypass downloading gems from rubygems.org just like pip has ways to bypass downloading Python packages from PyPI. But, in full honesty, I don’t have enough Ruby expertise to recommend a good way of doing this with URBANopt CLI.

What I can suggest is that, once the gems have been downloaded to the project folder the first time, they will remain there for future URBANopt simulations with that project folder. So you won’t have to download another ~0.5 GB of gems the next time you run the simulation (as you have already observed). This means that, if you have one URBANopt project folder with successfully downloaded gems, you can copy the .bundle file to other URBANopt project folders that you have and this will prevent the download from happening in those project folders:

Here is a zipped version of the .bundle file from my machine:

If you unblock it, unzip it and copy it into the URBANopt project folder you are trying to simulate, this should hopefully prevent further gem downloads from happening for that folder. Let us know if that works.

ZhengrongTao · May 14, 2025, 4:39am

Thanks @chris

After my testing, I am convinced that this can solve some of the problems. However, it is still inevitable that some dependencies need to be pulled. The execution of these pull processes will still bring me back to the problem I initially raised.

Mingbo provided me with the complete list of dependencies required for URBANopt to be pulled. However, during the actual execution process, the version pulled was different from the content listed in the list.

For instance, in this report, an attempt is still being made to pull in the existing dependencies. Moreover, the pulled-in addressable version is 2.8.6, but it should actually be 2.8.1. In the file you provided, the version of the “addressable” module is 2.8.1.

Defaulting bundle_install_path to bundle's local config value: 'D:\Desktop\test sample\.bundle\install'
Bundle final path is set to: D:/Desktop/test sample/.bundle/install
Defaulting bundle_install_path to bundle's local config value: 'D:\Desktop\test sample\.bundle\install'
Using runner options from runner.conf file
Initializing runner with dirname: 'D:/Desktop/test sample' and options: {:file_version=>"0.1.0", :max_datapoints=>1000000000, :num_parallel=>31, :run_simulations=>true, :verbose=>false, :gemfile_path=>"", :bundle_install_path=>"D:\\Desktop\\test sample\\.bundle\\install"}
@bundle_without_string = ''
bundler config exists
needs_config = false
needs_platform = true
Error running command: 'bundle lock --add_platform ruby'
stdout: Fetching source index from http://rubygems.org/
stderr: Network error while fetching
http://rubygems.org/quick/Marshal.4.8/addressable-2.8.6.gemspec.rz
(Net::ReadTimeout)
needs_update = true

The version of URBANopt I am currently using is 1.0.1, and the version of PollinationGHinstaller is 1.57.7. These are consistent with the version used in the latest urbanopt_Gemfile.

chris · May 15, 2025, 9:23pm

Hey @ZhengrongTao ,

I am not sure that I follow where adressable version 2.8.6 is coming up. Version 2.8.1 is what is in the .zip file of the .bundle that I uploaded here and it’s the same as what is listed in the urbanopt_Gemfile:

github.com

ladybug-tools/honeybee-openstudio-gem/blob/master/lib/files/urbanopt_Gemfile#L6


      
          source 'http://rubygems.org'
          
          ruby '~> 3.2.2'
          
          # pin these dependencies to prevent releases of gems from breaking URBANopt
          gem 'addressable', '2.8.1'
          gem 'ansi', '1.5.0'
          gem 'ast', '2.4.3'
          gem 'bcl', '0.8.0'
          gem 'builder', '3.2.4'
          gem 'docile', '1.4.1'
          gem 'faraday', '1.0.1'
          gem 'git', '1.12.0'
          gem 'json-schema', '4.3.1'
          gem 'json_pure', '2.8.1'
          gem 'macaddr', '1.7.2'

If you are using URBANopt 1.0.1 and you have unzipped the .bundle into the correct place, the URBANopt CLI stdout should move pretty quickly from the needs_platform step without downloading any new gems like so:

… but maybe there’s still some small call that is happening to rubygems.org. Am I correct that you can’t access Rubygems.org at all from your machine? Or is it just a slow internet connection?

ZhengrongTao · May 15, 2025, 9:42pm

hi,@chris

Yes, there might still be some unforeseen content being pulled from rubygems.org. In fact, it’s not entirely impossible to access rubygems.org. A long time ago, during the LBT 1.3 to 1.5 versions, I successfully used it to conduct some demonstrations. I guess it might be that the internet regulations in my area have become stricter again. So this resulted in him downloading at a very slow pace, but still he couldn’t complete the entire file transfer to my computer.

I believe I have successfully placed all the files you provided in the correct locations. I only modified the configuration files under the bundle file to point to the case folder that I am currently using.

In fact, just like Ladybug Tools uses the LB Versioner component for online installation and updates, in my area, updates or downloads cannot be performed without using a VPN.

At present, using the VPN method cannot completely solve the problems I am facing when using URBANopt. Is there a possibility that this pulling process can be observed in the operation report? Currently in URBANopt, this part is inactive and it might also be a point of confusion for many new users.

Perhaps we still need to have several more discussions on this issue. Currently, many users on our channel are experiencing similar problems.

chris · May 16, 2025, 4:53pm

Hey @ZhengrongTao ,

I have one more thing for you to try, which is admittedly a very hacky workaround and will only be a temporary solution until NREL releases of version of URBANopt CLI that does not require downloading gems for each project folder.

You can open the following dragonfly source code file in a text editor:

"C:\Program Files\ladybug_tools\python\Lib\site-packages\dragonfly_energy\writer.py"

… and edit this line of code in the file:

github.com

ladybug-tools/dragonfly-energy/blob/master/dragonfly_energy/writer.py#L121


      
              if len(folder) >= 60:
                  tr_len = 58 - len(folders.default_simulation_folder)
                  folder = os.path.join(folders.default_simulation_folder, sim_dir[:tr_len])
          else:
              assert len(folder) < 60, tr_msg.format(folder)
          
          # get rid of all simulation files that exists in the folder already
          dir_to_delete = ('hb_json', 'mappers', 'run')
          ext_to_delete = ('.bat', '.geojson', '.epw', '.mos')
          file_to_delete = (
              'Gemfile', 'Gemfile.lock', 'honeybee_scenario.csv', 'runner.conf',
              'simulation_parameter.json', 'system_params.json',
              'electrical_database.json', 'network.json'
          )
          if os.path.isdir(folder):
              files = os.listdir(folder)
              for f in files:
                  path = os.path.join(folder, f)
                  if os.path.isdir(path):
                      if f in dir_to_delete:
                          nukedir(path, True)

You’ll just remove 'Gemfile', 'Gemfile.lock', from the list there. Then save the file using admin privileges. Then, you can copy these two files into the project folder directory that contains the .bundle that you unzipped:

Gemfile.zip (2.3 KB)

If you do that, the next time that you run an URBANopt simulation out of that project folder, you should see the needs_platform step is false, which I think means that any use of the bundle commands is bypassed, meaning is should be theoretically possible to run an URBANopt simulation without an internet connection.

Granted, this has the potential to break things for people updating from older versions of URBANopt. So I don’t want to make a change like this an official part of the dragonfly plugin. But I think this should hopefully unblock you.

I had a call with NREL to discuss the challenges that are presented by the downloading from rubygems.org and they told me that they want to come up with a solution for it and put it in their funding requests for this year. So hopefully this will give us all some permanent fix without me having to engineer some major way to work around the whole URBANopt CLI on the dragonfly end.

chris · May 16, 2025, 4:58pm

Ok, yea, I just did a test on my end and confirmed that, with the workaround above, I was able to run an URBANopt simulation without any internet connection.

Granted, I still got an error printed to stdout that “bundle update” failed. But then it launches into a successful set of EnergyPlus simulations that complete:

So this is really the best that I can offer for the time being.