Custom pipeline stages with python

Here's a little guide on howto start making your own pipeline stages in python.

1. Create a configuration xml file for the stage in esp\etc\processors
2. Create a python code file for the processor in esp\lib\python2.3\processors
3. Restart the config server to have the GUI include the new stage (nctrl restart configserver)
4. Reset psctrl to compile the python to a pyc (psctrl reset)
5. Sometimes 4. is not enough to build the pyc, so I also use (nctrl restart procserver_1 procserver_2 ... procserver_x) to make another build attempt.

Setting max hits (maxoffset) in FAST esp

Been working on a little app that I use to compare two collections. I use it after updates to see if I've broken anything...

When a collection contains more that 10000 docs FAST will only return the first 10020 docs. Here is how to configure that limit:

Files:

• $FASTSEARCH/etc/config_data/RTSearch/webcluster/fdispatch.addon
• $FASTSEARCH/etc/config_data/QRServer/webcluster/etc/qrserver/qrserverrc
• $FASTSEARCH/etc/topfdispatchrc (if applicable)

are to be edited on search and QRServer nodes. Value maxoffset is to be set and processes qrserver, topfdispatch and search-1 restarted.

no doc procs registered to process a batch with priority 0

I came across this error message recently and I was able to figure out the problem.

WARNING    Could not send batch to ESP content distributor, will retry automatically. Reason given: process() failed: exception (no_resources) no doc procs registered to process a batch with priority 0

The problem was found in the document processor log files. (var\log\procserver\)

It turned out that a file was missing in esp\var\procserver\\\xxx.

What we did in order to solve the problem was:

1. stop the indexingdispather

2. stop all proc servers

3. delete all files and folders under esp\var\procserver\

4. start the indexingdispatcher

5. start proc servers