Loading...

modwsgi@googlegroups.com

[Prev] Thread [Next]  |  [Prev] Date [Next]

Re: [modwsgi] Independent PYTHONHOME for daemon process in separate process groups? Bill Freeman Thu Apr 05 05:00:56 2012

These changes work as desired and expected, producing sys.path the
same as when starting python from the activated virtual environment.
I like it.  I hope that it makes it to release.

Bill

On Sat, Mar 31, 2012 at 3:03 AM, Graham Dumpleton
<[EMAIL PROTECTED]> wrote:
> I have made changes in mod_wsgi source code repository for 4.0 which
> adds python-home option to WSGIDaemonProcess.
>
> The way it works is that if you do not specify WSGIPythonHome or
> python-home option to WSGIDaemonProcess, then the Python installation
> that mod_wsgi was compiled against will be used for everything.
>
> If you do not set WSGIPythonHome but set python-home option on a
> specific WSGIDaemonProcess group, then just that daemon process group
> will be initialised with that Python installation and everything else
> will use the Python installation that mod_wsgi was compiled against.
>
> If you set WSGIPythonHome then it will instead be used for everything,
> except for the case where a specific WSGIDaemonProcess directive has
> an overriding python-home option.
>
> That it acts as an overriding thing is why went with python-home after
> all, even though the Python shared library is always sourced from the
> Python installation mod_wsgi was compiled against.
>
> Now, this only works this way if WSGILazyInitialization is left as its
> default, which is On. If you turn off lazy initialisation for Python
> interpreters, which you shouldn't, because it causes memory leaks in
> Apache parent process due to Python not cleaning up after itself
> properly, then any python-home options are ignored completely and will
> always be Python installation mod_wsgi was compiled against, or any
> specific one set by WSGIPythonHome.
>
> How does one use this, is a simple matter of doing the following.
>
>  virtualenv --no-site-packages /some/path/venv
>
> Then set for a specific WSGIDaemonProcess directive, python-home to
> /some/path/venv. This is the same value as you would get if you ran in
> the interpreter:
>
>  import sys
>  print sys.prefix
>
> except that virtual environment sys.prefix isn't necessarily
> normalised properly.
>
> $ /some/path/venv/bin/python
> Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
> [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import sys
>>>> print sys.prefix
> /some/path/venv//bin/..
>>>> os.path.normpath(sys.prefix)
> '/some/path/venv'
>
> What this replaces in way things currently done is that previously you
> would also create:
>
>  virtualenv --no-site-packages /some/path/venv-baseline
>
> and set WSGIPythonHome to that directory, with python-path option to
> daemon process group being then being set to site-packages directory
> of the original virtual environment.
>
>  /some/path/venv/lib/python2.6/site-packages
>
> Now you do not worry about the baseline virtual environment, nor set
> WSGIPythonHome or python-path. Instead the only thing you need to do
> is set python-home to the location of the virtual environment.
>
> Bill, as the person who asked for it, can you give it a good testing. :-)
>
> Graham
>
> On 31 March 2012 09:13, Graham Dumpleton <[EMAIL PROTECTED]> wrote:
>> As explained in:
>>
>> http://code.google.com/p/modwsgi/wiki/VirtualEnvironments
>>
>> you would create a virgin Python virtual environment, with
>> --no-site-packages if old virtualenv version. You would point
>> WSGIPythonHome at that virgin Python virtual environment. It should be
>> dissociated from site-packages of system wide Python.
>>
>> You then create Python virtual environments for each daemon process
>> group, still with --no-site-packages, and set python-path option to
>> the site-packages directory of them.
>>
>> So, multi layers as alluded to.
>>
>> If python-virtualenv option were implemented WSGIPythonHome and need
>> to have virgin Python virtual environment layer would be skipped, and
>> python-path replaced with reference to per daemon process group Python
>> virtual environment root.
>>
>> When using the Python virtual environment with WSGIPythonHome as
>> described above what specifically is still being inherited from
>> site.py that is causing problems.
>>
>> The issue of sys.prefix is a contentious and confusing issue. For
>> virtualenv as standard part of Python 3.X, my understanding is that
>> sys.prefix might remain the parent Python installation and there will
>> be a separate variable indicating the location of the virtual
>> environment.
>>
>> Graham
>>
>> On 31 March 2012 08:58, Bill Freeman <[EMAIL PROTECTED]> wrote:
>>> On 3/30/12, Graham Dumpleton <[EMAIL PROTECTED]> wrote:
>>>> On 31 March 2012 05:55, Bill, KE1G <[EMAIL PROTECTED]> wrote:
>>>>> Might there already be a way to have per process group PYTHONHOME
>>>>> settings?
>>>>>
>>>>> And/or, please tell me that/why it wouldn't help.
>>>>>
>>>>> My purpose is to have several separate Django instances,
>>>>> each serving in a separate VirtualHost, and each using a
>>>>> separate virtualenv, all be run from a single apache.
>>>>>
>>>>> We are currently running 5 servers, a front end which does
>>>>> the virtual hosting, proxying to the other four separate apaches
>>>>> (same executable, separate configuration files, so linked against
>>>>> the same libpython.so).
>>>>>
>>>>> I was very pleased to discover the WSGIPythonHome directive
>>>>> recently, since now my sys.path looks correct (and my .wsgi
>>>>> scripts are much simpler).
>>>>>
>>>>> My understanding of WSGIDaemonProcess is that it spawns a
>>>>> separate process in which to run python interpreters, and the root
>>>>> apache, then proxys for them.  Further, I hope that if I put them in
>>>>> separate WSGIProcessGroup, that I can arrange to have traffic for
>>>>> a particular VirtualHost served by a particular process group (by
>>>>> putting the WSGIScriptAlias, the WSGIDaemonProcess, and the
>>>>> WSGIProcessGroup directives all within the VirtualHost element?).
>>>>>
>>>>> If all this is true, and if the python interpreter hasn't already been
>>>>> started by the time the daemon processes are forked, then it
>>>>> would be very nice to be able to set the PYTHONHOME env
>>>>> according to which VirtualHost's WSGIScriptAlias will use the
>>>>> daemon process (group).
>>>>>
>>>>> I did start to play with WSGIPythonHome, but discovered that it
>>>>> is not allowed inside of a VirtualHost.  Similarly, while there is a
>>>>> python-path= option for WSGIDaemonProcess, there is not one
>>>>> called, for example, python-home= .
>>>>>
>>>>> So:
>>>>>  1. Would this scheme work if I could set PYTHONHOME per
>>>>> daemon process group?
>>>>
>>>> It is not clear why it is needed when other mechanisms may achieve
>>>> what you want.
>>>
>>> Pointer please?
>>>
>>>>
>>>>>  2. Is there already a way to do this?
>>>>
>>>> No.
>>>>
>>>>>  3. Would adding a python-home= WSGIDaemonProcess option
>>>>> requre major refactoring?
>>>>
>>>> With the way mod_wsgi used to work, which was that Python interpreter
>>>> initialisation was done prior to fork of daemon processes, yes.
>>>>
>>>> By default it now only initialises interpreter after fork of process,
>>>> so technically it may be possible, I would need to look. It would have
>>>> to some how be disabled though when lazy initialisation of
>>>> interpreters had been turned off for some reason. That is, old way of
>>>> initialising in parent process before fork had been reenabled.
>>>>
>>>>>  4. Is this a bad idea even if possible?
>>>>
>>>> It could technically be useful, but using python-path to refer to the
>>>> site-packages directory of a second Python virtual environment
>>>> probably achieves the same result.
>>>
>>> Using python path doesn't quite achieve the same result, since the
>>> various things that the root python's home's site.py has added to
>>> sys.path are there (yes, I can remove them in the .wsgi script, but
>>> that makes things messy and brittle) and sys.prefix doesn't point
>>> to the virtualenv (though that probably doesn't matter).
>>>
>>>>
>>>> A python-home would would sort of eliminate one level of Python
>>>> installation, but not really, as the Python shared library is still
>>>> going to be loaded on mod_wsgi being loaded into Apache. It therefore
>>>> is still going to be sourced from the Python installation mod_wsgi was
>>>> compiled for and so all Python virtual environments still need to be
>>>> for same version of Python.
>>>
>>> I understand.  All the virtualenv instances are already using the same
>>> python (even though we are running 5 apache's, they all share the
>>> same modules - only the configurations differ).
>>>>
>>>> To make it clearer that daemon process groups wouldn't be truly using
>>>> everything from the virtual environment still, rather than
>>>> python-home, could be python-virtualenv option and would be given root
>>>> of the Python virtual environment.
>>>>
>>>> Graham
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To post to this group, send email to [EMAIL PROTECTED]
>>>> To unsubscribe from this group, send email to
>>>> [EMAIL PROTECTED]
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/modwsgi?hl=en.
>>>>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "modwsgi" group.
>>> To post to this group, send email to [EMAIL PROTECTED]
>>> To unsubscribe from this group, send email to [EMAIL PROTECTED]
>>> For more options, visit this group at 
>>> http://groups.google.com/group/modwsgi?hl=en.
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"modwsgi" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/modwsgi?hl=en.