mod_python

[Prev] Thread [Next]  |  [Prev] Date [Next]

Re: [mod_python] Multiple Interpreters mod_python optimization Graham Dumpleton Fri May 15 04:00:25 2009

2009/5/15 zsi <[EMAIL PROTECTED]>:
> Hey *
>
> I am creating a SOAP service that is based on ZSI and mod_python. I have
> the service running and it works, but it's not performing well, so I made
> did some output to the syslog from inside the handler:
>
> reloadchecker.py:
> ----------------
> testvar=1
> ----------------
>
> My handler:
> ----------------
> import capa0_9.soap.sessionservice as sessionservice
> import capa0_9.soap.zsi_handler as zsi_handler
> from mod_python import apache
> import capa0_9.tools.log as log
> import os,reloadchecker
>
> mod = __import__('encodings.utf_8', globals(), locals(), '*')
> mod = __import__('encodings.utf_16_be', globals(), locals(), '*')
>
> def handler(req):
>
>        """
>        The PythonHandler entry-point. This function must be present in
> order to have mod_apache dispatch a python operation.
>        """
>        zsi_handler.AsHandler(modules=(sessionservice,), request=req)
>        log.debug("REQUESTINFO: Interpreter Name = %s" %
> str(req.interpreter))
>        log.debug("REQUESTINFO: os.getpid()      = %s" % str(os.getpid()))
>        log.debug("REQUESTINFO: reloadchecker    = %s" %
> str(reloadchecker.testvar))
>        reloadchecker.testvar += 1
>        return apache.OK
> -----------------
>
> I log the interpreter name, which according to section 4.1 should make
> mod_python boost performance by keeping the interpreter alive as long as
> the apache server processes. And I log the pid of the interpreter. As it is
> clear to see in the output below a new pid is spawned each time, it does
> not reuse the interpreter and therefore, I guess, it has to reload
> everything for each request. Note that i also list the apache2 processes
> before and after I bombard the server with requests, it shows that the same
> processes are running before and after. I also made a dummy module
> "reloadchecker" that contains one variable "testvar". If it is reloaded the
> variable is initialized to 1, if it was already loaded it should be
> increased by 1 for each request.
>
> Am I missing something,

Yes.

> shouldn't mod_python optimize performance by
> reusing previously loaded modules from earlier requests?

Only if the request is actually handled by the same process. Since
Apache is a multiprocess web server, the requests could be handled by
any of the processes. You will obviously incur the load cost of the
interpreter and application for the first request against each
processor. Subsequent requests for a process should use the fact that
stuff is already loaded. You likely didn't do enough requests to have
touched all process and have a request actually go back to a process
which was already used.

Read:

  http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel

Since you appear to be using prefork MPM, also read about the dangers
of prefork MPM in:

  http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

If you want better control over how many processes are used for your
application and particularly so you can limit number to small fixed
number and/or use threading as well, then look at mod_wsgi daemon mode
instead.

BTW, the process IDs in your 'ps' output don't even match what your
Python code is producing. Looks like you aren't even talking to the
same Apache instance. Alternatively you have got some really silly
Apache configuration setup, such a MaxRequestsPerchild set to 1. But
even in that case, you would have expected some of those initial
process to have been recycled.

So, try and validate your configuration and whether you might have
multiple Apache installations running. Look for 'httpd' processes as
well.

Graham

> [EMAIL PROTECTED]:/srv/capa/www/0.9$ sudo ps aux | grep apache2
> mikrov    7396  0.0  0.1   3004   764 pts/2    S+   11:11   0:00 grep
> apache2
> root     22343  0.0  1.0  13728  5336 ?        Ss   May14   0:26
> /usr/sbin/apache2 -k start
> root     22346  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22347  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22348  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22349  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22350  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22383  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22392  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22407  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22716  0.0  0.6  13728  3136 ?        S    May14   0:03
> /usr/sbin/apache2 -k start
> root     23356  0.0  0.6  13728  3136 ?        S    May14   0:03
> /usr/sbin/apache2 -k start
>
> [EMAIL PROTECTED]:/srv/capa/www/0.9/sessionservice$ sudo tail -f
> /var/log/syslog | grep REQUESTINFO
> May 15 11:52:03 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:03 capa-server capa: REQUESTINFO: os.getpid()      = 7953
>
> May 15 11:52:03 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> May 15 11:52:03 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:03 capa-server capa: REQUESTINFO: os.getpid()      = 7954
>
> May 15 11:52:03 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> May 15 11:52:04 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:04 capa-server capa: REQUESTINFO: os.getpid()      = 7955
>
> May 15 11:52:04 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> May 15 11:52:05 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:05 capa-server capa: REQUESTINFO: os.getpid()      = 7959
>
> May 15 11:52:05 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> May 15 11:52:06 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:06 capa-server capa: REQUESTINFO: os.getpid()      = 7960
>
> May 15 11:52:06 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> May 15 11:52:07 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:07 capa-server capa: REQUESTINFO: os.getpid()      = 7961
>
> May 15 11:52:07 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:08 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:08 capa-server capa: REQUESTINFO: os.getpid()      = 7963
> May 15 11:52:08 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:08 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:08 capa-server capa: REQUESTINFO: os.getpid()      = 7964
> May 15 11:52:08 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:09 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:09 capa-server capa: REQUESTINFO: os.getpid()      = 7966
> May 15 11:52:09 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:10 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:10 capa-server capa: REQUESTINFO: os.getpid()      = 7968
> May 15 11:52:10 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:11 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:11 capa-server capa: REQUESTINFO: os.getpid()      = 7969
> May 15 11:52:11 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:12 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:12 capa-server capa: REQUESTINFO: os.getpid()      = 7970
> May 15 11:52:12 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:13 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:13 capa-server capa: REQUESTINFO: os.getpid()      = 7972
> May 15 11:52:13 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:13 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:13 capa-server capa: REQUESTINFO: os.getpid()      = 7973
> May 15 11:52:13 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:14 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:14 capa-server capa: REQUESTINFO: os.getpid()      = 7974
> May 15 11:52:14 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:15 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:15 capa-server capa: REQUESTINFO: os.getpid()      = 7976
> May 15 11:52:15 capa-server capa: REQUESTINFO: reloadchecker    = 1
> May 15 11:52:16 capa-server capa: REQUESTINFO: Interpreter Name =
> /srv/capa/www/0.9/sessionservice/
> May 15 11:52:16 capa-server capa: REQUESTINFO: os.getpid()      = 7977
> May 15 11:52:16 capa-server capa: REQUESTINFO: reloadchecker    = 1
>
> [EMAIL PROTECTED]:/srv/capa/www/0.9$ sudo ps aux | grep apache2
> mikrov    7457  0.0  0.1   3004   764 pts/2    S+   11:12   0:00 grep
> apache2
> root     22343  0.0  1.0  13728  5336 ?        Ss   May14   0:26
> /usr/sbin/apache2 -k start
> root     22346  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22347  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22348  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22349  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22350  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22383  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22392  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22407  0.0  0.6  13728  3136 ?        S    May14   0:04
> /usr/sbin/apache2 -k start
> root     22716  0.0  0.6  13728  3136 ?        S    May14   0:03
> /usr/sbin/apache2 -k start
> root     23356  0.0  0.6  13728  3136 ?        S    May14   0:03
> /usr/sbin/apache2 -k start
>
> Best regards
> Jakob Simon-Gaarde
>
> _______________________________________________
> Mod_python mailing list
> [EMAIL PROTECTED]
> http://mailman.modpython.org/mailman/listinfo/mod_python
>

_______________________________________________
Mod_python mailing list
[EMAIL PROTECTED]
http://mailman.modpython.org/mailman/listinfo/mod_python