tag:blogger.com,1999:blog-32235958040022850012024-03-06T06:26:41.507+01:00Paste hereWeb development with Python and PloneBertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-3223595804002285001.post-14451109565729121202009-06-02T16:52:00.003+02:002009-06-08T13:46:14.817+02:00Clean up trailing whitespaces in sourcesMy editor (emacs) is configured to remove trailing whitespaces in python files when I save them. This way I never commit modifications related to whitespaces changes, making the diffs readable since they contain relevant modifications only.<br /><br />Unfortunately not everyone do that, and when it comes to contributing to an existing project it can be very difficult to produce readable patches: sometimes while the actual patch is just a one line change the diff will show dozens of blank changes, due to whitespaces clean up.<br /><br />Diff has a switch to ignore whitespace changes, but this is incompatible with python: if you change the block level (indentation) it is just ignored.<br /><br />To clean up all python files found under a directory I use a shell one-liner:<br /><pre>$ find . -name '*.py' -exec sed -i {} -e 's/[ \t]*$//' ';'</pre>As usual it worked-for-me™ but comes with no warranty.<br /><br />My workflow for contributing a clean patch is like this:<br /><ol><li>create a local branch with bazaar, mercurial or git. Of course it depends if the project is already using one of them, but if you branch from a subversion repository it's just a matter of preferences<br /></li><li>clean whitespaces and commit (locally)</li><li>create and submit the patch normally<br /></li></ol>And here is the related configuration part for emacs:<br /><pre>;; whitespace cleanup<br />(defun my-py-no-trailing-space ()<br />; this hook is buffer local, can't add it globaly<br />(add-hook 'write-contents-functions 'delete-trailing-whitespace)<br />; if enabled, clear buffer at load time (this will automatically put the buffer in modified state, it might be annoying)<br />;(whitespace-cleanup)<br />)<br /><br />(add-hook 'python-mode-hook 'my-py-no-trailing-space)<br /></pre>I believe that other editors and environments (including Eclipse) can be configured for this, too.Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com7tag:blogger.com,1999:blog-3223595804002285001.post-19305657258582290212009-05-29T14:54:00.015+02:002009-06-01T12:43:25.850+02:00Installing Django, Solr, Varnish and Supervisord with BuildoutHere I'll detail my buildout configuration for an install of <a href="http://www.djangoproject.com/">Django</a>, <a href="http://lucene.apache.org/solr/">Solr</a> (http search server, ), <a href="http://varnish.projects.linpro.no/">Varnish</a> (http cache), and <a href="http://supervisord.org/">supervisord</a> for controling solr and varnish. I'll show how to get a Debian init script for supervisord (of course instructions are valid for Ubuntu). I'll detail parts of base.cfg for each service, and I'll try to explain what and why.<br /><br />This may not be the best way to do that but at least it works for me(tm), so I think it deserves to be shared.<br /><br />My buildout doesn't handle (yet?!) the apache configuration, so I will not cover this. For the curious here is the intended http chain:<br /><ul><li>Apache listens on port 80 and forwards requests to Varnish on port 3128. Unlike a typical Zope setup I don't need rewrite rules (here), simple proxying is enough</li><li>Varnish reaches the backend (django) on port 8000</li><li>Apache listens on port 8000, and serves Django with wsgi. Hopefully it serves only localhost.</li><li>django may query solr on port 8983<br /></li></ul><h2>Buildout files organisation</h2>In my buildout directory I have:<br /><ul><li>base.cfg: this file contains the core configuration. Specific settings (for developpement, production...) are made in files that extends base.cfg.</li><li>templates/: this directory contains file templates used in my buildout, for example I put here the template of the supervisord init script</li><li>varnish-conf/, solr-conf/: I'm versionning configuration for theses services, since the configurations generated by the recipes needed adjustements<br /></li></ul>Here is the "buildout" part in base.cfg:<br /><pre>[buildout]<br />newest = false</pre>newest: by default I don't want to check if eggs can be updated<br /><pre>versions = versions</pre>I want to have an exact version for a given egg, It will be declared in "versions" section. For example one can set "my.app = 1.0.3" (If you are developping "my.app" you can unset this in dev.cfg by declaring "my.app = ")<br /><pre>parts =<br /> svn-products<br /> django<br /> solr-files<br /> solr<br /> solr-conf<br /> varnish-build<br /> varnish<br /> supervisor<br /> supervisord_init_script<br /></pre>parts are installed in order. I'll detail them in time.<br /><pre>find-links =<br /> http://dist.repoze.org/<br /></pre>A distribution of <a href="http://www.pythonware.com/products/pil/">PIL</a> can be found here (it is poorly referenced at pypi). Also if you cannot upload "my.app" at pypi (customer project anyone?) and you don't have an egg server you can put egg tarballs of "my.app" at a local web server and put the link in "find-links".<br /><pre>eggs =<br /> PIL<br /> lxml<br /> psycopg2<br /> django-extensions<br /> django-cachepurge<br /> Werkzeug<br /> my.app<br /><br />[versions]<br />djangorecipe = 0.17.4<br />django-extensions = 0.4<br />Werkzeug = 0.5<br /></pre><h2>Django</h2>Parts related to django are "svn-products" and "django". "svn-products" allows me to get solango (not yet egg released :-( ).<br /><pre>[svn-products]<br />recipe = iw.recipe.subversion<br />urls =<br /> http://django-solr-search.googlecode.com/svn/trunk/solango solango<br /></pre>The django part. Note that I'm using django 1.0.2, since 1.1 is not yet released final. This is a matter of choice. Django 1.0.2 is available as an egg, but yet the recipe doesn't use this and downloads itself django.<br /><pre>[django]<br />recipe = djangorecipe<br />version = 1.0.2<br />control-script = django<br />wsgi = true<br />projectegg = my.app<br />eggs = ${buildout:eggs}<br />extra-paths = ${svn-products:location}<br /></pre><h2>Solr</h2>solr installation is made of 4 parts:<br /><ul><li>solr-files: download and unpack solr distribution:<br /><pre>[solr-files]<br />recipe = hexagonit.recipe.download<br />url = ftp://mir1.ovh.net/ftp.apache.org/dist/lucene/solr/1.3.0/apache-solr-1.3.0.tgz<br />md5sum = 23774b077598c6440d69016fed5cc810<br />strip-top-level-dir = true<br /></pre></li><li>solr: creates a runable instance of solr<br /><pre>[solr]<br />recipe = collective.recipe.solrinstance<br />solr-location = ${buildout:parts-directory}/solr-files<br />host = localhost<br />port = 8983<br /><br />unique-key = uniqueID<br />default-search-field = text<br /><br />index =<br /> name:uniqueID type:string indexed:true stored:true required:true<br /> name:text type:string indexed:true stored:true required:false omitnorms:false multivalued:true<br /></pre></li><li>solr-conf: I have added this to overwrite some config files in solr instance directory<br /><pre>[solr-conf]<br />recipe = iw.recipe.cmd<br />on_install = true<br />on_update = true<br />cmds =<br /> cp -v ${buildout:directory}/solr-conf/jetty.xml ${solr:jetty-destination}<br /> cp -v ${buildout:directory}/solr-conf/schema.xml ${solr:schema-destination}<br /> cp -v ${buildout:directory}/solr-conf/stopwords_fr.txt ${solr:schema-destination}<br /></pre>Why? because:<br /><ul><li>for jetty.xml I made solr listen only on localhost, this was not by default. If you choose to customize jetty.xml you must change absolute paths by relative ones. For example for "RequestLog", the path must be changed to: "../../var/solr/log/jetty-yyyy_mm_dd.request.log"</li><li>For schema.xml it is a bit different. The first times I have let the recipe generate it, but solango offers to output fields definitions from you application. Thus there is no reason to maintain them in buildout (in "solr" part). The command is:<br /></li></ul><pre>bin/django solr --fields --path=/tmp</pre>Then update schema.xml with the output.<br /></li><br /><li>solr-rebuild: "command" for reindexing django content (clear & rebuild)<br /><pre>[solr-rebuild]<br />recipe = iw.recipe.cmd<br />on_install = true<br />on_update = true<br /><br /># since solr is not started by solr-instance but supervisord, solr-instance has<br /># no pid file and thinks that solr is down. Thus we must run it with<br /># solr-instance to be able to "solr-instance purge"<br />cmds =<br /> ${buildout:bin-directory}/supervisorctl stop solr<br /> cp -v ${buildout:directory}/solr-conf/schema.xml ${solr:schema-destination}<br /> ${buildout:bin-directory}/solr-instance start<br /> COUNT=15; echo "Waiting $COUNT s"; sleep $COUNT<br /> ${buildout:bin-directory}/solr-instance purge<br /> time ${buildout:bin-directory}/${django:control-script} solr --reindex --batch-size 100<br /> ${buildout:bin-directory}/solr-instance stop<br /> ${buildout:bin-directory}/supervisorctl start solr<br /></pre>Actually I could have made a template of a shell script with collective.recipe.template, and I'll probably change for that solution; I made this quickly and I didn't know yet about the possibilities of the template recipe. Right now to rebuild solr-index I have to type:<br /><pre>$ bin/buildout install solr-rebuild</pre>Note that solr-rebuild part is not listed in buildout:parts, because I don't want to run it by default.</li></ul><h2>Varnish</h2>Nothing really advanced here. I have just customized varnish configuration to change a few things, and to add a ping url (important for supervisord).<br /><pre>[varnish-build]<br />recipe = zc.recipe.cmmi<br />url = http://downloads.sourceforge.net/varnish/varnish-2.0.4.tar.gz<br /><br />[varnish]<br />recipe = plone.recipe.varnish<br />daemon = ${varnish-build:location}/sbin/varnishd<br />bind = 127.0.0.1:3128<br />config = ${buildout:directory}/varnish-conf/varnish.vcl<br />telnet = localhost:8888<br />cache-size = 1G<br /><br /># foreground is needed for supervisor to control varnish correctly<br />mode = foreground<br /></pre>How to add a ping url? in varnish.vcl, at the beginning of vcl_recv:<br /><pre> # This url will always reply 200 whenever varnish is running<br />if (req.request == "GET" && req.url ~ "/varnish-ping") {<br />error 200 "OK";<br />}<br /></pre>For this I must admit I made a (very) quick search on the net; if anyone has a better solution please let me know!<br /><h2>Supervisor</h2><pre>[supervisor]<br />recipe = collective.recipe.supervisor<br />port = localhost:9001<br />user = admin<br />password = admin<br />plugins =<br /> superlance<br /><br /># solr security settings: see<br /># http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup<br />programs =<br /> 10 varnish (startsecs=10) ${buildout:directory}/bin/varnish true<br /> 20 solr (startsecs=10) java [-Djava.security.egd=file:/dev/urandom -jar start.jar] ${buildout:parts-directory}/solr true<br /><br />eventlisteners =<br /> SolrHttpOk TICK_60 ${buildout:bin-directory}/httpok [-p solr -t 20 http://localhost:8983/solr/]<br /> VarnishHttpOk TICK_60 ${buildout:bin-directory}/httpok [-p varnish -t 20 http://localhost:3128/varnish-ping]<br /></pre>For programs I set "startsecs" to 10 seconds. This tells supervisor to wait 10 seconds before considering that the program is properly running. This is important if your services take a bit of time before properly serving: if an event listeners is ran and finds a failure it may ask supervisor to restart again the service (i.e. before the service could ever complete its startup).<br /><br />Solr is not started with "bin/solr-instance fg", mainly because I needed to pass an aditionnal parameter (without it solr startup time was very long, from 1 to 5 min...).<br /><br />The event listeners are configured to check varnish and solr every minute. They order to restart them if they fail to answer.<br /><h3>Supervisor Init script for Debian</h3><pre>[supervisord_init_script]<br />recipe = collective.recipe.template<br />input = templates/supervisord_init.in<br />output = ${buildout:bin-directory}/supervisord_rc<br /></pre>For making "templates/supervisord_init.in" I copied /etc/init.d/skeleton and edited it. Important: do "chmod +x templates/supervisord_init.in", the permission will be reported on the generated file. Here is the diff:<br /><br /><pre>--- /etc/init.d/skeleton 2009-03-31 11:01:55.000000000 +0200<br />+++ templates/supervisord_init.in 2009-05-26 16:45:24.000000000 +0200<br />@@ -1,31 +1,31 @@<br />#! /bin/sh<br />### BEGIN INIT INFO<br />-# Provides: skeleton<br />+# Provides: supervisord<br /># Required-Start: $remote_fs<br /># Required-Stop: $remote_fs<br /># Default-Start: 2 3 4 5<br /># Default-Stop: 0 1 6<br />-# Short-Description: Example initscript<br />+# Short-Description: initscript for supervisord at ${buildout:bin-directory}<br /># Description: This file should be used to construct scripts to be<br /># placed in /etc/init.d.<br />### END INIT INFO<br /><br />-# Author: Foo Bar <foobar@baz.org><br />+# Author: Bertrand Mathieu <your_email@provider.tld><br />#<br />-# Please remove the "Author" lines above and replace them<br />-# with your own name if you copy and modify this script.<br />-<br /># Do NOT "set -e"<br /><br /># PATH should only include /usr/* if it runs after the mountnfs.sh script<br />PATH=/sbin:/usr/sbin:/bin:/usr/bin<br />-DESC="Description of the service"<br />-NAME=daemonexecutablename<br />-DAEMON=/usr/sbin/$NAME<br />-DAEMON_ARGS="--options args"<br />-PIDFILE=/var/run/$NAME.pid<br />+DESC="Start/Stop supervisord at ${buildout:bin-directory}"<br />+NAME=supervisord<br />+DAEMON=${buildout:bin-directory}/$NAME<br />+DAEMON_ARGS=""<br />+PIDFILE=${buildout:directory}/var/$NAME.pid<br />SCRIPTNAME=/etc/init.d/$NAME<br /><br />+# file owner will be used to run daemon<br />+OWNER=$(stat -c %U $DAEMON)<br />+<br /># Exit if the package is not installed<br />[ -x "$DAEMON" ] || exit 0<br /><br />@@ -48,9 +48,9 @@<br /># 0 if daemon has been started<br /># 1 if daemon was already running<br /># 2 if daemon could not be started<br />- start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \<br />+ start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --chuid $OWNER --test > /dev/null \<br /> || return 1<br />- start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \<br />+ start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --chuid $OWNER -- \<br /> $DAEMON_ARGS \<br /> || return 2<br /># Add code here, if necessary, that waits for the process to be ready<br />@@ -68,7 +68,7 @@<br /># 1 if daemon was already stopped<br /># 2 if daemon could not be stopped<br /># other if a failure occurred<br />- start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME<br />+ start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --chuid $OWNER --name $NAME<br />RETVAL="$?"<br />[ "$RETVAL" = 2 ] && return 2<br /># Wait for children to finish too if this is a daemon that forks<br />@@ -77,7 +77,7 @@<br /># that waits for the process to drop all resources that could be<br /># needed by services started subsequently. A last resort is to<br /># sleep for some time.<br />- start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON<br />+ start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --chuid $OWNER --exec $DAEMON<br />[ "$?" = 2 ] && return 2<br /># Many daemons don't delete their pidfiles when they exit.<br />rm -f $PIDFILE<br />@@ -93,7 +93,7 @@<br /># restarting (for example, when it is sent a SIGHUP),<br /># then implement that here.<br />#<br />- start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME<br />+ start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --chuid $OWNER --name $NAME<br />return 0<br />}<br /></your_email@provider.tld></foobar@baz.org></pre>Notes:<br /><ul><li>the daemon is run by the owner of bin/supervisord. Most of the time it is the user who has ran buildout (hopefully it is not root!)</li><li>I have used a bash construct to get the owner ("OWNER=$(stat -c %U $DAEMON)"), this could be changed for pure sh<br /></li><li>thus bin/supervisord_rc (start | stop) can be run by this user, without the need for "sudo". Without this "solr-rebuild" could not work.<br /></li></ul>To install it in init.d:<br /><pre>$ cd /etc/init.d<br />$ sudo ln -s /path/to/buildout/bin/supervisord_rc my_preferred_service_name<br />$ sudo updated-rc.d my_preferred_service_name defaults<br /></pre>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com10tag:blogger.com,1999:blog-3223595804002285001.post-62742202158971958922009-05-25T17:30:00.007+02:002009-05-25T18:23:38.646+02:00django-cachepurge 0.1aI have released django-cachepurge 0.1a. It is available as an egg for an easy installation with buildout or virtualenv+easy_install, for example.<br /><br />This package allows django to purge HTTP cache when a model instance is changed or deleted. It does this by sending asynchronous "PURGE" requests to one or more upstream HTTP cache (such as Squid or Varnish). It is inspired by <a href="http://plone.org/products/cachefu">Plone CacheFu components</a> (more specifically: CMFSquidTool).<br /><br />Unfortunatly Django does not have a "post_commit" signal (it would be the best place to do such a job), so purge requests are sent when response has been computed: if an exception occurs during response the urls are not purged. This is done by the middleware.<br /><br />Pre-requisite: the cache must be configured to accept and handle "PURGE" requests from the server where the django application is hosted.<br /><h2>Configuration on django side:</h2><ol><li>The application must be the first app declared in settings.INSTALLED_APP. The reason is that it listens to the <a href="http://docs.djangoproject.com/en/dev/ref/signals/#class-prepared">class_prepared</a> signal to connect <a href="http://docs.djangoproject.com/en/dev/ref/signals/#post-save">post_save</a> and <a href="http://docs.djangoproject.com/en/dev/ref/signals/#post-delete">post_delete</a> handlers on eligible models (more on that below). If you put other app before django-cachepurge it may miss their models. Note that the package name uses an underscore.<br /><code>INSTALLED_APPS = (<br /> 'django_cachepurge',<br /> ...<br /> )<br /></code><br /></li><li>add "django_cachepurge.middleware.CachePurge" in settings.MIDDLEWARE_CLASSES</li><li>define settings.CACHE_URLS to the cache root for django. CACHE_URLS can be a single string or an iterable of strings. For example:<br /><code>CACHE_URLS = 'http://127.0.0.1:3128'</code><br /></li></ol><h2>How urls are found?</h2>If the model has a get_absolute_url method, this url will be purged. Additionnaly you can define "get_purged_urls": it should return a list of urls. This is useful for "through" models used in M2M relation to invalidate url of linked contents for example. If the model has none of these methods, nothing happens (the signals are not connected).<br /><br />Pypi: <a href="http://pypi.python.org/pypi/django-cachepurge/">http://pypi.python.org/pypi/django-cachepurge/</a><br />Launchpad: <a href="http://launchpad.net/django-cachepurge/">http://launchpad.net/django-cachepurge/</a>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com3tag:blogger.com,1999:blog-3223595804002285001.post-70364584738139450392009-02-10T14:29:00.007+01:002009-02-10T14:54:08.699+01:00Defining and accessing macros located in browser:page templateThe case: I want to define a simple browser page (let's name it "mypage") for a page template where I defined some metal macros (in my case it is a template for an archetypes field). My product does not provide a skin for portal_skins, I don't want to add a layer and all the generic setup stuff just for a single template. I'm using plone 3.1.<br /><br />The problem: @@mypage/macros does not work as legacy portal_skins page templates used to.<br /><br />Solution: define a simple class like this:<br /><pre>from Products.Five import BrowserView<br /><br />class MacrosView(BrowserView):<br /><br />@property<br />def macros(self):<br />return self.index.macros<br /></pre>The ZCML for "mypage":<br /><pre><browser:page<br />for="*"<br />name="mypage"<br />class=".macros.MacrosView"<br />template="mypage.pt"<br />allowed_attributes="macros"<br />permission="zope.Public" /><br /></pre>There could be a better, less verbose solution (like providing a meta definition for zcml, in order to avoid declaring "class" and "allowed_attributes"). We could also patch Five BrowserView.<br /><br />In my case I have been able to use mypage as a template for my archetypes widget:<br /><pre>MyWidget(macro="@@mypage",)</pre>Compared to legacy PT you will loose some builtins (like python: test()), but that kind of logic should be (easily) moved into a dedicated view class. This is noticeable in the case you are customizing an old template (like archetypes/widgets/file.pt ;-))<br /><br />Dunno if it is the "right way of doing things", at least it worked-for-me(tm).Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2tag:blogger.com,1999:blog-3223595804002285001.post-77363400070073389432009-01-21T11:30:00.006+01:002009-01-21T16:53:10.030+01:00Useful script in a plone developer toolboxSometimes something weird happen on the production site and you have to investigate on data from that site because you can't reproduce the problem on development site. When it's really hard you have to copy the Data.fs and run a separate instance to work on it. What I'm putting here is a script that changes all users passwords to their id, and also change email property: this allows to login as anybody easily, and no mail can be sent to the actual users. All you have to do is create a "Script (Python)" in ZMI at portal root, put this code and click "test". It's not a revolution, it's not-so-good-practice(tm), it's just a convenience ;-)<br /><pre>mtool = context.portal_membership.aq_inner<br />pu = context.plone_utils.aq_inner<br />acl = context.acl_users.aq_inner<br />count = 0<br /><br />for uid in acl.getUserIds():<br /> count += 1<br /> acl.userSetPassword(uid, uid)<br /> member = mtool.getMemberById(uid)<br /> pu.setMemberProperties(member, email='me.the.developper@mydomain.tld')<br /> print uid<br /><br />print<br />print count, "users"<br />return printed<br /></pre>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com3tag:blogger.com,1999:blog-3223595804002285001.post-72613830718896145042009-01-02T15:00:00.010+01:002009-01-02T15:51:00.121+01:00Profiling made easyRecently I had to profile some pages on a Plone 2.5 (zope 2.9). I collected some datas on interesting pages with the help of the well-known <a href="http://www.dieter.handshake.de/pyprojects/zope/#bct_sec_4.8">ZopeProfiler 1.7.2</a> but I had to patch it to avoid <a href="http://plone.org/support/region/de#nabble-td1344811">an error</a>:<pre>--- ZopeProfiler.py~ 2007-06-26 10:43:25.000000000 +0200<br />+++ ZopeProfiler.py 2008-12-22 18:02:26.000000000 +0100<br />@@ -393,10 +393,10 @@<br /># Five broke 'getPhysicalPath' for its view classes -- work around<br />try: p= gP()<br />except:<br />- _log.error("calling 'getPhysicalPath' failed for %r", s,<br />- exc_info=sys.exc_info()<br />- )<br />- return<br />+ # _log.error("calling 'getPhysicalPath' failed for %r", s,<br />+ # exc_info=sys.exc_info()<br />+ # )<br />+ return ('?', _Empty, fn)<br />if type(p) is StringType: fi= p<br />else: fi= '/'.join(p)<br />return (fi,_Empty,fn)<br /></pre>A few years ago we had no other option than digging in the raw stats as they come from Stats.sort_stats().print_stats(). Since then <a href="http://tarekziade.wordpress.com/2008/08/25/visual-profiling-with-nose-and-gprof2dot/">there is a new tool</a>: <a href="http://code.google.com/p/jrfonseca/wiki/Gprof2Dot">Gprof2dot</a>. The author also made something more than handy: <a href="http://code.google.com/p/jrfonseca/wiki/XDot">xdot.py</a>.<br /><br />Now just add a little bash function:<br /><pre>$ function build_dot() { ./gprof2dot.py -f pstats -o $(basename $1 .pstats).dot $1; }</pre>Then my workflow for profiling some pages could be faster and easier:<br /><ol><li>Profile a page, and save "some_page.pstats"</li><li>run "build_dot some_page.pstats"</li><li>run "./xdot.py some_page.dot"</li><li>visit the graph<br /></li></ol>Here is the first overview:<br /><div style="text-align: left;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXm2UHcv4oiUGAqe_Hx-w2tPDxmmfkjVJg1lNpP7oCfTryb2ir4C8__1t24s9kHz_DSWZZ-j8iTCrS6Pkqc6h7SBqziX9073YM3HtKgKfsP4Dk3QmeDmD8feOgifWngJqSAMo-wMGijD4s/s1600-h/graph-stats-overview.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 148px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXm2UHcv4oiUGAqe_Hx-w2tPDxmmfkjVJg1lNpP7oCfTryb2ir4C8__1t24s9kHz_DSWZZ-j8iTCrS6Pkqc6h7SBqziX9073YM3HtKgKfsP4Dk3QmeDmD8feOgifWngJqSAMo-wMGijD4s/s320/graph-stats-overview.png" alt="" id="BLOGGER_PHOTO_ID_5286702966839475538" border="0" /></a>The mouse wheel allows to zoom in/out, holding left-click and moving the mouse will move the graph. It's quite easy to quickly find some hotspots, sometimes they will appear very obviously:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHk9qOd7PCWFcK-kjqhmrt2RtewgAsYbQOQ0BKPsM0lQQiNW5wUYTqSQOXwBr67ZAWM4QW_IByFPjfomDkZlnItywxQ0vdDKNgrz3McSAW2AkdihT4bFPE8Gives93BTVCOTNuwqySwwa4/s1600-h/schema-copy-hotspot.png"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 141px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHk9qOd7PCWFcK-kjqhmrt2RtewgAsYbQOQ0BKPsM0lQQiNW5wUYTqSQOXwBr67ZAWM4QW_IByFPjfomDkZlnItywxQ0vdDKNgrz3McSAW2AkdihT4bFPE8Gives93BTVCOTNuwqySwwa4/s320/schema-copy-hotspot.png" alt="" id="BLOGGER_PHOTO_ID_5286704136306417026" border="0" /></a><br /></div>I can read: 69% of total time spent in schema copy. In this particular case I know there is just one object with a "Schema" method, so probably it would be a good idea to review the code here to reduce the number of schema copies, or thinking about adding some cache decorator if it's possible (like plone.memoize). The graph does not tell what to do, though ;-)<br /><br />Another interesting hotspot (in plone 2.5): for some pages up to 15% of the time in spent in... getAllowedTypes (just 1 call - nearly 11% in pythonproducts.py __bobo_traverse__).Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com5tag:blogger.com,1999:blog-3223595804002285001.post-20935725005381854362009-01-02T11:52:00.008+01:002009-01-02T12:14:32.740+01:00GenericSetup and dependence on circular dependencies problemAs of Plone 3.1.6 there is a problem with import step dependencies: if you register a custom import step through zcml, and if this step depends on "portlets", "content" or "plone-final", then your import step will be inserted <span style="font-weight: bold;">before</span> its dependencies. This is because local steps (i.e. defined in an import_steps.xml file) are listed after ZCML ones, and in its final loop GS ordering method will insert remaining steps as they comme.<br /><br />The big problem is when you must to execute "mysite-final" after "portlets" for example.<br /><br />There is a <a href="http://dev.plone.org/plone/ticket/8350">related ticket</a> on plone.org, I have added a comment with a patch (and tests) for GS to deal better with this kind of dependencies. It may be useful <span style="font-weight: bold;">now</span> for someone. This ticket may not be the best place to put that, but sadly I really don't have the time to discuss it in the right mailing list.<br />Here is the idea: basically the final loop is modified to insert first any step involved in circular chain, and then it will try to insert remaining ones with dependency resolution. Thus "mysite-final" will always be inserted after "portlets".Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com0tag:blogger.com,1999:blog-3223595804002285001.post-59547497514116720542008-12-04T12:59:00.007+01:002008-12-04T13:54:18.602+01:00Managing Apache virtualhosts and squid with iw.recipe.squidI have a number of plone sites on a server, generally one site is associated with one zope instance. I am using buildout and <a href="http://pypi.python.org/pypi/iw.recipe.squid">iw.recipe.squid</a> to add easily a new virtualhost with squid proxying. Im my case the server is running Ubuntu server 8.04, but this should apply without change to a Debian server. This may look overkill for juste one or two sites, but in my case I may have up to 30 different sites (of course all zope instances are not on localhost!).<br /><br />Let's suppose I have these sites:<br /><ul><li>www.site1.net, zope instance on localhost (127.0.01) port 8080, plone path is located at /site1/portal<br /></li><li>www.someothersite.com, zope instance on 10.2.0.5 port 9080, plone path /somepath/portal</li></ul>buildout.cfg:<br /><br /><pre>[buildout]<br />parts = squid<br />versions = versions<br /><br />[versions]<br />iw.recipe.squid = 0.9<br /><br />[squid]<br />recipe = iw.recipe.squid<br />squid_owner = proxy<br />squid_visible_hostname = myservername<br />squid_cache_dir = /var/cache/squid<br />squid_log_dir = /var/log/squid<br /><br />squid_accelerated_hosts =<br /> www.site1.net: 127.0.0.1:8080/site1/portal<br /> www.someothersite.com: 10.2.0.5:9080/somepath/portal<br /></pre>After running buildout for the first time:<br /><ul><li>make a symbolic link "/etc/squid.conf" pointing to parts/squid/etc/squid.conf</li><li>run "bin/squidctl createswap" if required</li><li>check that squid starts normally and that helper processes are running, too (iRedirector.py, squidAcl.py, squidRewriteRules.py)<br /></li></ul>After having added one or more site:<br /><ul><li>in /etc/apache2/sites-available create a symbolic link for all config files located in parts/squid/apache. In our case they should be named "vhost_www.site1.net_80.conf", etc</li><li>run a2ensite to active sites ("a2ensite vhost_www.site1.net_80.conf")</li><li>reload apache<br /></li></ul>Adding a new site will be just a matter of adding a new line in squid_accelerated_hosts, running buildout, making the symbolic links and reloading apache.<br /><br />As of<a href="http://pypi.python.org/pypi/iw.recipe.squid"> iw.recipe.squid</a> 0.9, squid and apache log reside in the same directory, but the next release should allow to have different directory; it should also allow you to set "combined" rather than "common" log format for apache.<br /><br />Of course do not forget to configure properly CacheSetup on every plone site.Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2tag:blogger.com,1999:blog-3223595804002285001.post-79447405447275982882008-10-16T18:21:00.006+02:002008-10-16T18:56:02.717+02:00How to change add permision of another product contentIn a customer project I have had to use CalendarX. CalendarX uses "Add portal content" for content add permission. But they wanted that only "Manager" be able to add a calendar. I could have gone straightforward by overriding "getNotAddableTypes" script in portal_skins, but I chose to try to patch the product from mine (let's call it "my.product").<br /><br />I first tried to import CalendarX and change the permission name with a simple monkey patch, but I immediately had to dig into zope initialization process: "my.product" is imported after Products.CalendarX, because "my.product" gets loaded by Products.Five initialisation, and Five is loaded after CalendarX.. so my patch arrives too late in the init process. What to do? try harder! I found I could reload the product after having patched the permission.<br /><br />Honestly I don't know if it can raise issue, and I would prefer something that looks less "hacky", but...<br />So, here is the guilty code!<br /><pre># this is my/product/__init__.py<br />import logging<br />LOG = logging.getLogger(__name__)<br /><br />def initialize(context):<br />"""Initializer called when used as a Zope 2 product."""<br /><br />from Products import CalendarX<br />cxf_perm = "%s: Add CalendarX content" % (PROJECT_NAME,)<br />setDefaultRoles(cxf_perm, ('Manager',))<br />CalendarX.DEFAULT_ADD_CONTENT_PERMISSION = cxf_perm<br />CalendarX.config.DEFAULT_ADD_CONTENT_PERMISSION = cxf_perm<br /><br /># also patch this, else it will try to register the profile twice<br /># and the registry will complain<br />from Products.GenericSetup.registry import ProfileRegistry<br />dummy_registry = ProfileRegistry()<br />std_registry = CalendarX.profile_registry<br />CalendarX.profile_registry = dummy_registry<br /><br /># FIXME: is there anything cleaner to get app?<br />app = context._ProductContext__app<br />from OFS import Application<br />Application.reinstall_product(app, 'CalendarX')<br />CalendarX.profile_registry = std_registry<br />LOG.info("patched CalendarX.DEFAULT_ADD_CONTENT_PERMISSION: use '%s'"<br /> % cxf_perm)<br /></pre>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2tag:blogger.com,1999:blog-3223595804002285001.post-72119201882607075422008-09-22T11:08:00.004+02:002008-09-22T12:06:34.316+02:00iw.eggproxy 0.2.0We have released <a href="http://pypi.python.org/pypi/iw.eggproxy">iw.eggproxy 0.2.0</a>. Bugs fixed:<br /><ul><li>package index/download files: skip modules installed in local system (resulted<br /> in copying a directory instead of downloading a file)</li><li>update script crashed with invalid/obsolete package name</li><li>get eggs distributions for all versions/platforms, instead of system ones<br /></li><li>malformed tag in generated indexes</li></ul>We are going to rename it to <span style="font-style: italic;">collective.eggproxy</span>. The next release will provide a standalone server and (hopefully) a WSGI application. This should make it easier to start with it.<br /><br />A sprint occured this summer in the topic, a few months after iw.eggproxy was released, and some people chose to create a full mirorring tool called <a href="http://pypi.python.org/pypi/z3c.pypimirror">z3c.pypimirror</a>.<br />Here are some differences with <a href="http://pypi.python.org/pypi/z3c.pypimirror">z3c.pypimirror</a>:<br /><ul><li><span style="font-style: italic;">eggproxy</span> relies on setuptools. It sees what "easy_install" can see: no more, no less. Also it does not include its own machinery to read pypi indexes and follow links.<br /></li><li><span style="font-style: italic;">eggproxy</span> provides on-demand any egg provided at pypi. You don't need to know in advance what packages you will need, you don't need to download them before: just ask for them as if you were directly on pypi server</li><li>OTOH if the server has network problems to reach pypi <span style="font-style: italic;">eggproxy</span> may not be able to serve an egg, where <span style="font-style: italic;">z3.pypimirror</span> would have already downloaded it. This case happens only if <span style="font-style: italic;">eggproxy</span> has not already served once the egg.</li><li><span style="font-style: italic;">z3c.pypimirror</span> creates a static directory layout suitable for any HTTP server; because of its nature <span style="font-style: italic;">eggproxy</span> needs <span style="font-style: italic;">apache</span> and<span style="font-style: italic;"> mod_python</span> (next release should change this strict requirement by providing a standalone service and/or WSGI).</li></ul>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com0tag:blogger.com,1999:blog-3223595804002285001.post-37522640555718033672008-06-24T18:39:00.004+02:002008-06-24T18:53:18.287+02:00Five 1.5.6 testbrowser and recent version of mechanizeDoing development with Plone 3.1, I encountered a KeyError when I tried to use Five.testbrowser.Browser:<br /><pre>Browser()<br />KeyError: '_seek'</pre>I eventually found that my system (currently Ubuntu 8.04) had the package "python-mechanize" 0.1.7b, which masks the one provided by Zope 2.10 (0.1.2b). Unfortunatly there has been API changes between 0.1.2b and 0.1.7b... (see classes UserAgent/UserAgentBase)<br /><br />The solution is to tell buildout to get mechanize egg 0.1.2b: this way it will mask the system library.<br /><pre>[buildout]<br />versions = versions<br />eggs=<br /> ...<br /> mechanize<br /><br />[versions]<br />mechanize = 0.1.2b<br /></pre>Et voila!Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2tag:blogger.com,1999:blog-3223595804002285001.post-18890364643130571082008-06-09T17:41:00.002+02:002008-06-09T18:02:25.902+02:00Buildout, ploneldap and ldap products (2)In my <a href="http://zebert.blogspot.com/2008/04/buildout-ploneldap-and-ldap-products.html">previous post</a> I described how to get a buildout part to get PloneLDAP with an up-to-date LDAPUserFolder. Things are even simpler now since all involved products have been released as eggs. So just list theses in your buildout along with other eggs:<br /><br />eggs =<br /> ...<br /> <a href="http://pypi.python.org/pypi/Products.LDAPUserFolder/">Products.LDAPUserFolder</a><br /> <a href="http://pypi.python.org/pypi/Products.LDAPMultiPlugins/">Products.LDAPMultiPlugins</a><br /> <a href="http://pypi.python.org/pypi/Products.PloneLDAP/">Products.PloneLDAP</a><br /><br />Second important thing: LDAPUserFolder 2.9 has been released. If you used 2.9-beta you must upgrade, since an important bug related to the negative cache has been fixed. PloneLDAP documents usage with LDAPUserFolder 2.8, but 2.9 is definitely worth it: The negative cache feature avoids doing too many ldap request, especially useless/unsuccessful ones! Many thanks to <span>Jens Vagelpohl (<a href="http://www.dataflake.org/software/ldapuserfolder">http://www.dataflake.org/software/ldapuserfolder</a>).<br /></span>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com1tag:blogger.com,1999:blog-3223595804002285001.post-64628138459145632872008-06-09T15:53:00.003+02:002008-06-09T16:11:24.356+02:00Quick workflow migration from plone 2.x to plone 3When migrating a site from plone 2.x to plone 3, I often have to port customized workflows. Back in the plone 2.0 days we hadn't generic setup, and workflows were created with python code (with the help of DCWorkflowDump), or directly created in the ZMI and installed on final site by importing a zexp.<br /><br />Nowadays we want to use generic setup (GS).<br /><br />The simplest way I have found is to export the workflow as a zexp, import this zexp into the new site, and then make a GS export. Done! you've got a nice XML definition of your workflow, ready to be included in your product GS profile.<br /><br />Once I encountered one caveat: if the original workflow contains strings that are not in plain ascii (in titles, etc...), the GS export will fail. You'll have to relabel properly everywhere, and add all those labels (msgids) to your translation files (I think i18ndude does it for you).Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com0tag:blogger.com,1999:blog-3223595804002285001.post-57882100901882523192008-06-06T16:36:00.004+02:002008-06-06T16:45:59.397+02:00iw.eggproxy: a proxy for pypiToday we have released <a href="http://pypi.python.org/pypi/iw.eggproxy">iw.eggproxy</a>. It's a module for apache mod_python. Its purpose is to serve as a <a href="http://pypi.python.org/">pypi</a> proxy.<br /><br />The first motivation to make it was because we have had to work in a private network with a very very slow internet access: for example updating a plone buildout could take more than one hour (just checking eggs freshness) when it should be no more than a few minutes. This condition also prevent to run some kind of rsync against pypi. So the obvious solution was to proxy the eggs we need, on-demand.<br /><br />The module is installed as a handler on a <a href="http://httpd.apache.org/docs/2.2/mod/core.html#location">Location</a>. When accessing this location, eggproxy will serve an index similar to pypi simple view.<br /><br />This is done like this:<br /><ul><li>we already have the information in index.html: just let apache serve the file</li><li>or, we use setuptools to fetch index information, build index.html, and let apache serve the file<br /></li></ul>This allows eggproxy to serve all available eggs from pypi, without actually having to download the whole pypi + content.<br /><br />Then easy_install can see a package is available on our server, and tries to fetch information on available eggs:<br /><ul><li>the subdirectory and its index.html already exists: just let apache serve the file</li><li>or, we use setuptools again to get package information, make the subdirectory (package name) and build index.html<br /></li></ul><br />Finally, when trying to fetch an egg we do the same:<br /><ul><li>the file is already present and is served by apache</li><li>or we use setuptools to get the file on the server<br /></li></ul>iw.eggproxy also provides an update script: "eggproxy_update". This script refreshes the main index and all proxied eggs, if their index.html is older than the interval specified in the configuration file (24h by default).<br /><br />We have installed it here:<a href="http://release.ingeniweb.com/pypi.python.org-mirror"> http://release.ingeniweb.com/pypi.python.org-mirror</a><br /><br />Known bugs: some packages on pypi don't have eggs, then eggproxy does not respond. This is the case with "reportlab" for example.<br /><br />Enhancements:<br /><ul><li>indexes aggregation. At ingeniweb we plan to install it on local server and agregate pypi and some private eggs indexes.</li><li>Standalone/pluggable server. Currently we are bound to apache + mod_python, which may not suit to anyone.<br /></li></ul>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com1tag:blogger.com,1999:blog-3223595804002285001.post-60920462634225513572008-04-22T14:36:00.004+02:002008-06-09T18:03:38.732+02:00Buildout, ploneldap and ldap products<span style="font-weight: bold;">UPDATE: </span>all products are available as eggs. See <a href="http://zebert.blogspot.com/2008/06/buildout-ploneldap-and-ldap-products-2.html">this post</a>.<br /><br /><a href="http://plone.org/products/ploneldap">PloneLDAP</a> is a great product, but its bundle ships a not so recent version of <a href="http://www.dataflake.org/software/ldapuserfolder">LDAPUserFolder</a>. <a href="http://www.dataflake.org/software/ldapuserfolder">LDAPUserFolder</a> and <a href="http://www.dataflake.org/software/ldapuserfolder">LDAPMultiPlugins</a> download urls are both ending with "/download". That seems to confuse plone.recipe.distros, because it installs just one of them. Furthermore, at plone.org PloneLDAP single product (as opposed to PloneLDAP bundle) download url ends with... a space! (%20).<br />So we have uploaded theses tarballs on release.ingeniweb.com. Here is a working part for buildout:<br /><pre><br />[ldap_products]<br />recipe = plone.recipe.distros<br />urls =<br />http://release.ingeniweb.com/third-party-dist/LDAPUserFolder-2.9-beta.tgz<br />http://release.ingeniweb.com/third-party-dist/LDAPMultiPlugins-1.5.tgz<br />http://release.ingeniweb.com/third-party-dist/PloneLDAP-1.0.tar.gz<br /></pre>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com4tag:blogger.com,1999:blog-3223595804002285001.post-44920728440250980202008-02-11T14:56:00.000+01:002008-02-11T15:13:13.241+01:00iw.rejectanonymous: private site with plone 3.0We have made a small package to provide the functionality described <a href="http://zebert.blogspot.com/2008/01/making-private-site-with-plone-30.html">in my previous post</a>. It is named "<a href="http://pypi.python.org/pypi/iw.rejectanonymous">iw.rejectanonymous</a>".<br /><br />Quick recipe to use it from an integration product (i.e a product responsible of setting up a plone site for your particular environment/customer/...):<br /><br /><ul><li>Add in configure.zcml:<pre><include package="iw.rejectanonymous" /></pre></li><br /><li>Add python code to activate it for your site. This is probably done in a function called by generic setup, this is often located in setuphandlers.py:<br /><pre>from zope.interface import alsoProvides<br />from iw.rejectanonymous import IPrivateSite<br /><br />def setupPortal(portal):<br />if not IPrivateSite.providedBy(portal):<br /> alsoProvides(portal, IPrivateSite)<br /></pre></li></ul>The second step can be done through the ZMI with the "Interfaces" tab.Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com0tag:blogger.com,1999:blog-3223595804002285001.post-67143697894532255402008-01-31T16:04:00.001+01:002008-04-13T13:02:29.407+02:00Making a private site with plone 3.0<span style="font-weight: bold;">UPDATE</span>: we have made a product for this: <a href="http://zebert.blogspot.com/2008/02/iwrejectanonymous-private-site-with.html">iw.rejectanonymous</a><br /><br />There is a <a href="http://plone.org/documentation/how-to/creating-private-plone-site">document at plone.org</a> suggesting to use plone 3.0 builtin "intranet" workflows, however this will not make a site absolutely private, i.e. force user to login before he can view anything. This is the use case for an extranet for example.<br /><br />In the past we used to put something like tal:define="dummy here/rejectAnonymous" in global_defines.pt, rejectAnonymous was a skin script. Now with the help of events we can do far better, and it will work for any content/object within a plone site. As a consequence we must be careful about what is allowed to be retrievied anonymously, since anonymous should be able to see a themed login page.<br /><br />The idea has been taken from <a href="http://svn.zope.de/plone.org/plone/plone.aftertraverse/trunk/">plone.aftertraverse</a>. An event is sent before traversal, but not immediatly after. The problem is that authentication is performed after traversal. Fortunately the request object accepts to register post traverse hooks, with arbitrary parameters.<br /><br />The code, zcml part:<br /><br /><pre><subscriber handler=".hooks.insertRejectAnonymousHook"<br /> for="Products.CMFCore.interfaces.ISiteRoot<br /> zope.app.publication.interfaces.IBeforeTraverseEvent"<br /> /></pre><br /><br />and hooks.py:<br /><br /><pre># -*- coding: utf-8 -*-<br />from zExceptions import Unauthorized<br /><br />valid_subparts = set(('login.js', 'spinner.gif',<br /> 'portal_css', 'portal_javascripts'))<br /><br />def rejectAnonymous(portal, request):<br /> mtool = portal.portal_membership<br /> if mtool.isAnonymousUser():<br /> url = request.physicalPathFromURL(request['URL'])<br /> if url and not (url[-1] in ('login_form', 'require_login')<br /> or [path for path in url<br /> if path in valid_subparts]):<br /> raise Unauthorized, "You must be authenticated"<br /><br /><br />def insertRejectAnonymousHook(portal, event):<br /> """<br /> """<br /> event.request.post_traverse(rejectAnonymous, (portal, event.request))<br /><br /></pre><br />The code checking for allowed path may not be the best, and it could certainly be more clever but for-me-it-worked(tm)Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com0tag:blogger.com,1999:blog-3223595804002285001.post-66209659288131764192008-01-09T16:38:00.000+01:002008-01-31T16:37:44.657+01:00VersionConflict in buildoutToday I tried to update my buildout from plone 3.0.4 to 3.0.5. In the relevant section I just changed:<br /><pre>- recipe = plone.recipe.plone==3.0.4<br />+ recipe = plone.recipe.plone==3.0.5<br /></pre><br />and re-run buildout, but I got a "VersionConflict" error:<br /><br /><pre>[...]<br />Uninstalling plone.<br />While:<br /> Installing.<br /> Uninstalling plone.<br /> Loading recipe 'plone.recipe.plone==3.0.4'.<br /><br />An internal error occured due to a bug in either zc.buildout or in a<br />recipe being used:<br /><br />VersionConflict:<br />(plone.recipe.plone 3.0.5 (/home/bmathieu/.buildout/eggs/plone.recipe.plone-3.0.5-py2.4.egg), Requirement.parse('plone.recipe.plone==3.0.4'))<br /></pre><br />The only way I found to get rid of this is to edit the hidden file named ".installed.cfg", and replace the line:<br /><pre>- recipe = plone.recipe.plone==3.0.4<br />+ recipe = plone.recipe.plone<br /></pre><br />Then buildout could finish its job. I don't know if this is clean, but it may help.Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2tag:blogger.com,1999:blog-3223595804002285001.post-34112743411057148682007-12-07T15:30:00.000+01:002007-12-11T17:14:17.994+01:00Maintenance mode for Apache<span style="font-size:85%;"><span style="font-style: italic;">This has been tested with Plone behind Apache but should apply to any site with Apache as a front server.<br /></span></span><span style="font-size:100%;"><span>In maintenance mode we want Apache to serve a single static page with its associated ressources, and have any other requests redirected to this page. The static files are:<br /></span></span><br /><ul><li>maintenance.html : the main file</li><li>ressources/ : directory containing CSS, javascript, images...</li><li>robots.txt (instruct to not index maintenance.html)<br /></li></ul><br /><span><span style="font-size:100%;"><span>To create a page with the same design of the real site the firefox extension </span></span></span><a id="gn15_6" href="https://addons.mozilla.org/fr/firefox/addon/4723" title="https://addons.mozilla.org/fr/firefox/addon/4723">Save Complete</a> is of great help.<br /><br /><span style="font-size:100%;">There is 2 strategies:<br /></span><ul><li><span style="font-size:100%;">Doing it dynamically by testing a parameter; in our case it will be the presence of a file</span></li><li><span style="font-size:100%;">With an alternative configuration file, swapped with the normal one at maintenance time<br /></span></li></ul><span style="font-size:100%;">In this example we are running a plone site on the same host, port 8080.</span><br /><br /><span style="font-size:85%;"><span style="font-size:130%;">1. Dynamically:</span></span><br /><br /><pre>DocumentRoot /static/files<br /><directory><br />Options FollowSymLinks<br />AllowOverride None<br />Order deny,allow<br />Allow from all<br />Satisfy all<br /><br />DirectoryIndex maintenance.html<br /></directory><br />RewriteEngine on<br /></pre><br />The next rule tests the presence of the regular file "/some/path/maintenance.txt"; if it exists we set the environment variable "MAINTENANCE" to 1.<br /><pre>RewriteCond /some/path/maintenance.txt -f<br />RewriteRule ^(.*)$ - [env=MAINTENANCE:1]<br /></pre>In maintenance mode we want the browser to never perform any cache: as soon as we'll return to production mode, the normal site should appear. Important: the <span style="font-style: italic;">headers</span> module must be enabled.<br /><br /><pre>Header set cache-control "max-age=0,must-revalidate,post-check=0,pre-check=0" env=MAINTENANCE<br />Header set Expires -1 env=MAINTENANCE<br /></pre><br />Next rule instructs to let pass any request matching maintenance files (maintenance.html + CSS/JS/images ressources), else redirect to the maintenance page.<br /><br /><pre>RewriteCond %{ENV:MAINTENANCE} 1<br />RewriteCond %{REQUEST_URI} ^/ressources [OR]<br />RewriteCond %{REQUEST_URI} ^/maintenance.html [OR]<br />RewriteCond %(REQUEST_URI) ^/robots.txt [OR]<br />RewriteRule .* - [L]<br /><br />RewriteCond %{ENV:MAINTENANCE} 1<br />RewriteCond %{REQUEST_URI} !^/maintenance.html<br />RewriteRule ^.* /maintenance.html [L,R]<br /></pre><br />The next and last rewrite rule is for normal mode.<br /><br /><pre>RewriteCond %{ENV:MAINTENANCE} !1<br />RewriteCond %{HTTP:Authorization} ^(.*)<br />RewriteRule ^(.*) http://localhost:8080/VirtualHostBase/http/%{HTTP_HOST}:80/plone/site/VirtualHostRoot/$1 [P]<br /></pre><br />Now, to put the site in maintenance mode, just do "touch /some/path/maintenance.txt"; delete maintenance.txt to go back in normal mode.<br /><span style="font-size:85%;"><span style="font-size:130%;"><blockquote></blockquote>2. Using an alternative configuration file:<br /><span style="font-size:100%;"><br />This case is much simpler. First create a static configuration:<br /></span></span></span><br /><pre>DocumentRoot /static/files<br />RewriteEngine on<br /><br />RewriteCond %{REQUEST_URI} ^/ressources [OR]<br />RewriteCond %{REQUEST_URI} ^/maintenance.html [OR]<br />RewriteCond %(REQUEST_URI) ^/robots.txt [OR]<br />RewriteRule .* - [L]<br /><br />RewriteCond %{REQUEST_URI} !^/maintenance.html<br />RewriteRule ^.* /maintenance.html [L,R]<br /><br />Header set cache-control "max-age=0,must-revalidate,post-check=0,pre-check=0"<br />Header set Expires -1<br /></pre><br /><br />On Debian this file must be placed in <span style="font-style: italic;">/etc/apache2/sites-available</span>. If the site configuration file is named "plone", you can name its maintenance counterpart "plone-maint" for example. To switch to maintenance mode:<br /><br />> sudo a2dissite plone<br />> sudo a2ensite plone-maint<br />> sudo /etc/init.d/apache reload<br /><br />To go back to normal mode just switch "plone" and "plone-maint".<br /><br /><span style="font-size:130%;">Enhancement not covered here:<br /><br /></span><span style="font-size:100%;">While the site is supposed to show a maintenance page, it may be desirable to let the maintainer access the site through apache: this should be done with a rewrite condition/rule placed first (condition based on Ip address for example).</span>Bertrand Mathieuhttp://www.blogger.com/profile/08397928766523129786noreply@blogger.com2