Pip install confusion?

edpmay · June 26, 2024, 8:19pm

This might be a question for the Rhino forum, but thought I’d try here first. I am having trouble pip installing to the ladybug_tools interpreter and wonder if anyone has a thought on how to solve?

I am trying to update some existing packages within my ladybug_tools installation, and so running the simple command:

"C:\Program Files\ladybug_tools\python\python.exe" -m pip install PHX from the Command Prompt (in Admin mode) it gives me the result:

Requirement already satisfied: PHX in c:\users\em\appdata\roaming\python\python310\site-packages (1.47.4)
Requirement already satisfied: honeybee-core>=1.58.22 in c:\users\em\appdata\roaming\python\python310\site-packages (from PHX) (1.58.23)
Requirement already satisfied: honeybee-energy>=1.106.9 in c:\users\em\appdata\roaming\python\python310\site-packages (from PHX) (1.106.10)
Requirement already satisfied: honeybee-ph>=1.28.15 in c:\users\em\appdata\roaming\python\python310\site-packages (from PHX) (1.28.15)
Requirement already satisfied: pydantic<2.0 in c:\program files\ladybug_tools\python\lib\site-packages (from PHX) (1.10.13)
Requirement already satisfied: PH-units>=1.5.9 in c:\users\em\appdata\roaming\python\python310\site-packages (from PHX) (1.5.11)
...

The part I don’t understand about that is that: sometimes it is finding an existing package installation in c:\users\{me}\appdata\roaming\python310\site-packages\... and then sometimes it finds the existing packages in c:\program files\ladybug_tools\python\lib\site-packages\....

As a result, the packages within the ladybug_tools are not getting updated properly? When I go to import them in Rhino, the older ones in the ladybug_tools are getting loaded, not the updated ones in the ...\users\... ?

Does anyone understand what is going on there?
Is there a way to tell it to ignore the global install and force an update to the packages within the ladybug_tools interpreter?
Is this related to the PYTHONHOME business?
Are both of these locations related the ladybug_tools installation? If so, why are there two locations?

Or I guess, maybe an even shorter question (cus’ maybe I don’t need to understand all the above…): “How can I pip install to the ladybug_tools python interpreter in a way that works reliably, consistently, without a ton of errors and permissions issues and confusion?”

any advice would be much appreciated.

@edpmay

chris · June 26, 2024, 8:43pm

Hi @edpmay ,

This question is highly specific to how we use Python in Ladybug Tools so I think you were correct to ask here first.

I do have an inference about what has happened on your machine since it has happened to me when run:

"C:\Program Files\ladybug_tools\python\python.exe" -m pip install [PACKAGE_NAME]

… from a command prompt that was not in Admin mode. It seems that pip has a mechanism that defaults to installing things in you AppData folder when it finds that it does not have access to edit the Python installation that is being called. I don’t know why pip does this but it can really make a mess of your Python packages since pip basically decides that the AppData folder is now the place where the package should live instead of the place that it could not edit. This is true even if you then open a new Command Prompt in Admin mode and try to update things. And, of course, Rhino is still looking to the location in Program Files to load the Python libraries and has no knowledge about your AppData folder.

There are ways to fix this from command line (pip uninstalling and then reinstalling using a Command Prompt in Admin mode has worked for me if it’s just one package). But I have often found it’s easier to just completely uninstall Ladybug Tools/Pollination when this happens and then I reinstall via the single-click installer.

It is not related to the PYTHONHOME business. And, no, your AppData Python packages are not related to the ladybug_tools installation. This is some mechanism in pip that we have no control over.

The best answer to this:

… is to always pip install packages either from a command prompt with Admin privleges or you could use the Ladybug.Executor.exe that we ship with the pollination installers to ensure the pip process is always running in Admin mode. You can see an example of how to do this in the source code of the LB Versioner component:

github.com

ladybug-tools/ladybug-grasshopper/blob/master/ladybug_grasshopper/src/LB Versioner.py#L64-L74


      
          executor_path = os.path.join(
              folders.ladybug_tools_folder, 'grasshopper', 'ladybug_grasshopper_dotnet',
              'Ladybug.Executor.exe')
          
          # run the command to update everything
          if os.name == 'nt' and os.path.isfile(executor_path) and \
                  'Program Files' in executor_path and not is_user_admin():
              cmd = [
                  executor_path, folders.python_exe_path,
                  '-m ladybug_rhino change-installed-version'
              ]

The GHE Designer component also has a good example that is specifically using a pip install command:

github.com

ladybug-tools/dragonfly-grasshopper/blob/master/dragonfly_grasshopper/src/DF GHE Designer.py#L179-L184


      
          install_cmd = 'pip install ghedesigner=={}'.format(GHE_DESIGNER_VERSION)
          if os.name == 'nt' and os.path.isfile(executor_path) and \
                  'Program Files' in executor_path:
              pip_cmd = [
                  executor_path, folders.python_exe_path, '-m {}'.format(install_cmd)
              ]

edpmay · June 26, 2024, 9:43pm

Thank you @chris , this is super helpful as always.

I will review the Executor - that might be just the right solution.

So is there ever a scenario where you would allow (require) the user to pip-install when not in admin mode? I honestly don’t even remember why I ever had that as an option in my installer… was there some install method that wanted you to be in non-admin for some reason?

We’re having tons of issues with folks that can’t get the HBPH packge to install reliably, so trying to figure out what we’re doing wrong and how we can do it better in the future.

-e

edpmay · June 26, 2024, 10:43pm

From what I can tell experimenting now, it appears that you are exactly right:

IF the user runs pip install ... from a Command Prompt (or, in my actual case, from within Rhino) while in admin-mode, everything works as expected and the packages are installed in the C:\Program Files\... location.
BUT, IF the user runs pip install ... from a Command Prompt while NOT in admin mode, python creates a new folder C:\Users\em\AppData\Roaming\Python\... all on its own without asking, and installs the packages over there.
Then, IF the user tries again to pip install ... in a Command Prompt. whether in or out of admin mode, pip will find the already existing packages over in C:\Users\... and so then not install/update them properly within the C:\Program Files\...

So as soon as the user does it wrong, even by accident, everything gets all mixed up from then on until that C:\Users\... folder is removed.

Does that summary appear accurate to you @chris? (I know I’m just re-stating what you already said… trying to make sure I understand it though…)

I think I know how to fix it now at least, if that is what is happening.

@edpmay

chris · June 27, 2024, 9:14pm

Glad this was helpful, @edpmay .

Well, I would never recommend forgoing Admin mode if the user has the option of using it. But I know from experience back in practice that some IT overlords are very strict and I would sometimes get blocked waiting for IT to grant Admin privileges to update my software. For those situations, people can have the ladybug tools installation in:

C:\Users\[USERNAME]\ladybug_tools

… and you technically don’t need Admin privleges to run pip in that location.

Granted, we are all generally moving away from this practice now that most IT departments recognize the Pollination single-click installers as safe and protocols for IT grating Admin priveleges are generally getting better. But that is the reason why you see some if/else statements in our source code for using the Executor.

Yes, your understanding all sounds correct to me.