-1

I have Ubuntu 9.10 server with 12G Ram/Quad Core/HD 80GB RAID. and i have installed solr lucidworks-enterprise-installer-1.7 on it to index a small database (about 20k articles).
once our editors start to use solr functionalities to search for a specific article (our editors who use solr are 5 editors only), server-load goes up exponentially!!

$ top
top - 08:29:29 up 25 days, 20:04,  1 user,  load average: 46.23, 59.69, 38.29
Tasks: 2695 total,   1 running, 2693 sleeping,   0 stopped,   1 zombie
Cpu(s): 14.2%us,  2.6%sy,  0.0%ni, 45.7%id, 35.9%wa,  0.1%hi,  1.4%si,  0.0%st
Mem:  12331880k total, 12245716k used,    86164k free,     3140k buffers
Swap:  3229024k total,  3228600k used,      424k free,   139184k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                     
17448 root      20   0 3710m 1.4g 3300 S   61 11.7  37:32.39 java     

EDIT
in my dataconfig.xml

<entity name="root" pk="id" preImportDeleteQuery="data_source:1" query="select a.id, a.body, a.headline  title ,a.date  datecreated, a.title_id, t.name  publisher_name from article as a inner join title as t on t.id=a.title_id" transformer="TemplateTransformer" deltaImportQuery="select a.id, a.body, a.headline  title ,a.date  datecreated, a.title_id, t.name  publisher_name from article as a inner join title as t on t.id=a.title_id where a.id='${dataimporter.delta.id}'" deltaQuery="select id from article where last_update_time > '${dataimporter.last_index_time}'">
      <field column="data_source" template="1"/>
      <field column="data_source_type" template="Jdbc"/>
      <field column="data_source_name" template="db_solr"/>
    </entity>

and the table in mysql database

mysql> desc article;
+------------------+--------------+------+-----+-------------------+-------+
| Field            | Type         | Null | Key | Default           | Extra |
+------------------+--------------+------+-----+-------------------+-------+
| id               | int(255)     | NO   | PRI |                   |       | 
| nid              | int(11)      | NO   | MUL | 0                 |       | 
| title_id         | int(11)      | NO   | MUL | 0                 |       | 
| language_id      | int(255)     | NO   | MUL |                   |       | 
| headline         | varchar(255) | YES  |     | NULL              |       | 
| summary          | text         | YES  |     | NULL              |       | 
| body             | text         | YES  |     | NULL              |       | 
| author           | varchar(255) | YES  |     | NULL              |       | 
| date             | date         | YES  | MUL | NULL              |       | 
| parsed_at        | datetime     | YES  | MUL | NULL              |       | 
| updated_at       | datetime     | YES  |     | NULL              |       | 
| last_update_time | timestamp    | NO   |     | CURRENT_TIMESTAMP |       | 
+------------------+--------------+------+-----+-------------------+-------+
12 rows in set (0.02 sec)

mysql> select count(*) from article;
+----------+
| count(*) |
+----------+
|    19560 | 
+----------+
1 row in set (0.00 sec)

also i found 7 instances of the line below by applying ps aux |grep "DlucidworksHome=/etc/solr" | grep -v grep

java -server -DlucidworksHome=/etc/solr -XX:MaxPermSize=256m -DSTOP.PORT=8887 -DSTOP.KEY=stopLucidWorks -Djava.awt.headless=true -Dlog4j.configuration=file:conf/log4j.xml -Dorg.restlet.engine.loggerFacadeClass=org.restlet.ext.slf4j.Slf4jLoggerFacade -Duser.language=en -Duser.country=US -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djetty.port=8888 -Djetty.home=jetty -jar ./jetty/start.jar ./jetty/etc/jetty.xml ./jetty/etc/jetty-ssl.xml

any idea about what might cause this... (solr takes 11.7% of 12G Ram!!). usually my load average is about 3-5 but once i start solr it becomes 40-70
am i wrong and solr is normal to make such load??
Please excuse my ignorance in solr :)
Thanks for your help

3
  • Can we see this exponential server load graph? Also, without knowing the actual size of your database, and the searches being performed, it's impossible to know whether this is normal, and without knowing your exact server configuration, it's impossible to give recommendations for improvement.
    – womble
    Aug 7, 2011 at 9:06
  • Hmm... not seeing any edits.
    – womble
    Aug 7, 2011 at 9:18
  • please check now Aug 7, 2011 at 9:21

1 Answer 1

0

(I'm posting this as an answer instead of comment because of the length)

We had Lucene at my last job, indexing well over 500.000 articles on a virtual server with 1 CPU and 3GB RAM..

I'm starting to think that something is terribly wrong with the way you're interacting against your articles.. is it perhaps doing a full index every time a single article gets updated/created? You have a very high amount of I/O wait on your CPU, wich is making me think that the database is killing your raid.

These are of course just guesses, but there is no way that this is a problem with your operating system.

You should probably contact solr, as this is a bit too narrow for a general Q/A site.

1
  • indeed i dont make full index at all. i just made it once and then i am using delta-import. however thank a lot for your attention Aug 7, 2011 at 12:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .