Re: [modwsgi] Independent PYTHONHOME for daemon process in separate process groups?

It is already in for mod_wsgi 4.0 release. :-)

No idea what I am doing about back porting for 3.4 though.

Graham

On 5 April 2012 21:30, Bill Freeman <[EMAIL PROTECTED]> wrote:
> These changes work as desired and expected, producing sys.path the
> same as when starting python from the activated virtual environment.
> I like it.  I hope that it makes it to release.
>
> Bill
>
> On Sat, Mar 31, 2012 at 3:03 AM, Graham Dumpleton
> <[EMAIL PROTECTED]> wrote:
>> I have made changes in mod_wsgi source code repository for 4.0 which
>> adds python-home option to WSGIDaemonProcess.
>>
>> The way it works is that if you do not specify WSGIPythonHome or
>> python-home option to WSGIDaemonProcess, then the Python installation
>> that mod_wsgi was compiled against will be used for everything.
>>
>> If you do not set WSGIPythonHome but set python-home option on a
>> specific WSGIDaemonProcess group, then just that daemon process group
>> will be initialised with that Python installation and everything else
>> will use the Python installation that mod_wsgi was compiled against.
>>
>> If you set WSGIPythonHome then it will instead be used for everything,
>> except for the case where a specific WSGIDaemonProcess directive has
>> an overriding python-home option.
>>
>> That it acts as an overriding thing is why went with python-home after
>> all, even though the Python shared library is always sourced from the
>> Python installation mod_wsgi was compiled against.
>>
>> Now, this only works this way if WSGILazyInitialization is left as its
>> default, which is On. If you turn off lazy initialisation for Python
>> interpreters, which you shouldn't, because it causes memory leaks in
>> Apache parent process due to Python not cleaning up after itself
>> properly, then any python-home options are ignored completely and will
>> always be Python installation mod_wsgi was compiled against, or any
>> specific one set by WSGIPythonHome.
>>
>> How does one use this, is a simple matter of doing the following.
>>
>>  virtualenv --no-site-packages /some/path/venv
>>
>> Then set for a specific WSGIDaemonProcess directive, python-home to
>> /some/path/venv. This is the same value as you would get if you ran in
>> the interpreter:
>>
>>  import sys
>>  print sys.prefix
>>
>> except that virtual environment sys.prefix isn't necessarily
>> normalised properly.
>>
>> $ /some/path/venv/bin/python
>> Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
>> [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import sys
>>>>> print sys.prefix
>> /some/path/venv//bin/..
>>>>> os.path.normpath(sys.prefix)
>> '/some/path/venv'
>>
>> What this replaces in way things currently done is that previously you
>> would also create:
>>
>>  virtualenv --no-site-packages /some/path/venv-baseline
>>
>> and set WSGIPythonHome to that directory, with python-path option to
>> daemon process group being then being set to site-packages directory
>> of the original virtual environment.
>>
>>  /some/path/venv/lib/python2.6/site-packages
>>
>> Now you do not worry about the baseline virtual environment, nor set
>> WSGIPythonHome or python-path. Instead the only thing you need to do
>> is set python-home to the location of the virtual environment.
>>
>> Bill, as the person who asked for it, can you give it a good testing. :-)
>>
>> Graham
>>
>> On 31 March 2012 09:13, Graham Dumpleton <[EMAIL PROTECTED]> wrote:
>>> As explained in:
>>>
>>> http://code.google.com/p/modwsgi/wiki/VirtualEnvironments
>>>
>>> you would create a virgin Python virtual environment, with
>>> --no-site-packages if old virtualenv version. You would point
>>> WSGIPythonHome at that virgin Python virtual environment. It should be
>>> dissociated from site-packages of system wide Python.
>>>
>>> You then create Python virtual environments for each daemon process
>>> group, still with --no-site-packages, and set python-path option to
>>> the site-packages directory of them.
>>>
>>> So, multi layers as alluded to.
>>>
>>> If python-virtualenv option were implemented WSGIPythonHome and need
>>> to have virgin Python virtual environment layer would be skipped, and
>>> python-path replaced with reference to per daemon process group Python
>>> virtual environment root.
>>>
>>> When using the Python virtual environment with WSGIPythonHome as
>>> described above what specifically is still being inherited from
>>> site.py that is causing problems.
>>>
>>> The issue of sys.prefix is a contentious and confusing issue. For
>>> virtualenv as standard part of Python 3.X, my understanding is that
>>> sys.prefix might remain the parent Python installation and there will
>>> be a separate variable indicating the location of the virtual
>>> environment.
>>>
>>> Graham
>>>
>>> On 31 March 2012 08:58, Bill Freeman <[EMAIL PROTECTED]> wrote:
>>>> On 3/30/12, Graham Dumpleton <[EMAIL PROTECTED]> wrote:
>>>>> On 31 March 2012 05:55, Bill, KE1G <[EMAIL PROTECTED]> wrote:
>>>>>> Might there already be a way to have per process group PYTHONHOME
>>>>>> settings?
>>>>>>
>>>>>> And/or, please tell me that/why it wouldn't help.
>>>>>>
>>>>>> My purpose is to have several separate Django instances,
>>>>>> each serving in a separate VirtualHost, and each using a
>>>>>> separate virtualenv, all be run from a single apache.
>>>>>>
>>>>>> We are currently running 5 servers, a front end which does
>>>>>> the virtual hosting, proxying to the other four separate apaches
>>>>>> (same executable, separate configuration files, so linked against
>>>>>> the same libpython.so).
>>>>>>
>>>>>> I was very pleased to discover the WSGIPythonHome directive
>>>>>> recently, since now my sys.path looks correct (and my .wsgi
>>>>>> scripts are much simpler).
>>>>>>
>>>>>> My understanding of WSGIDaemonProcess is that it spawns a
>>>>>> separate process in which to run python interpreters, and the root
>>>>>> apache, then proxys for them.  Further, I hope that if I put them in
>>>>>> separate WSGIProcessGroup, that I can arrange to have traffic for
>>>>>> a particular VirtualHost served by a particular process group (by
>>>>>> putting the WSGIScriptAlias, the WSGIDaemonProcess, and the
>>>>>> WSGIProcessGroup directives all within the VirtualHost element?).
>>>>>>
>>>>>> If all this is true, and if the python interpreter hasn't already been
>>>>>> started by the time the daemon processes are forked, then it
>>>>>> would be very nice to be able to set the PYTHONHOME env
>>>>>> according to which VirtualHost's WSGIScriptAlias will use the
>>>>>> daemon process (group).
>>>>>>
>>>>>> I did start to play with WSGIPythonHome, but discovered that it
>>>>>> is not allowed inside of a VirtualHost.  Similarly, while there is a
>>>>>> python-path= option for WSGIDaemonProcess, there is not one
>>>>>> called, for example, python-home= .
>>>>>>
>>>>>> So:
>>>>>>  1. Would this scheme work if I could set PYTHONHOME per
>>>>>> daemon process group?
>>>>>
>>>>> It is not clear why it is needed when other mechanisms may achieve
>>>>> what you want.
>>>>
>>>> Pointer please?
>>>>
>>>>>
>>>>>>  2. Is there already a way to do this?
>>>>>
>>>>> No.
>>>>>
>>>>>>  3. Would adding a python-home= WSGIDaemonProcess option
>>>>>> requre major refactoring?
>>>>>
>>>>> With the way mod_wsgi used to work, which was that Python interpreter
>>>>> initialisation was done prior to fork of daemon processes, yes.
>>>>>
>>>>> By default it now only initialises interpreter after fork of process,
>>>>> so technically it may be possible, I would need to look. It would have
>>>>> to some how be disabled though when lazy initialisation of
>>>>> interpreters had been turned off for some reason. That is, old way of
>>>>> initialising in parent process before fork had been reenabled.
>>>>>
>>>>>>  4. Is this a bad idea even if possible?
>>>>>
>>>>> It could technically be useful, but using python-path to refer to the
>>>>> site-packages directory of a second Python virtual environment
>>>>> probably achieves the same result.
>>>>
>>>> Using python path doesn't quite achieve the same result, since the
>>>> various things that the root python's home's site.py has added to
>>>> sys.path are there (yes, I can remove them in the .wsgi script, but
>>>> that makes things messy and brittle) and sys.prefix doesn't point
>>>> to the virtualenv (though that probably doesn't matter).
>>>>
>>>>>
>>>>> A python-home would would sort of eliminate one level of Python
>>>>> installation, but not really, as the Python shared library is still
>>>>> going to be loaded on mod_wsgi being loaded into Apache. It therefore
>>>>> is still going to be sourced from the Python installation mod_wsgi was
>>>>> compiled for and so all Python virtual environments still need to be
>>>>> for same version of Python.
>>>>
>>>> I understand.  All the virtualenv instances are already using the same
>>>> python (even though we are running 5 apache's, they all share the
>>>> same modules - only the configurations differ).
>>>>>
>>>>> To make it clearer that daemon process groups wouldn't be truly using
>>>>> everything from the virtual environment still, rather than
>>>>> python-home, could be python-virtualenv option and would be given root
>>>>> of the Python virtual environment.
>>>>>
>>>>> Graham
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "modwsgi" group.
>>>>> To post to this group, send email to [EMAIL PROTECTED]
>>>>> To unsubscribe from this group, send email to
>>>>> [EMAIL PROTECTED]
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/modwsgi?hl=en.
>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "modwsgi" group.
>>>> To post to this group, send email to [EMAIL PROTECTED]
>>>> To unsubscribe from this group, send email to [EMAIL PROTECTED]
>>>> For more options, visit this group at 
>>>> http://groups.google.com/group/modwsgi?hl=en.
>>>>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "modwsgi" group.
> To post to this group, send email to [EMAIL PROTECTED]
> To unsubscribe from this group, send email to [EMAIL PROTECTED]
> For more options, visit this group at 
> http://groups.google.com/group/modwsgi?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/modwsgi?hl=en.