Running DiscoverEd - Revision history

Akozak: /* Crawl */

2010-10-11T20:54:02Z

‎Crawl

2010-09-17T22:24:23Z

‎Crawl

2010-09-17T22:24:07Z

‎Crawl

2010-09-10T15:44:26Z

‎Deploying new WARs

2010-09-10T15:43:57Z

2010-09-08T22:50:11Z

2010-08-30T19:17:15Z

2010-08-30T19:10:06Z

‎Crawl

2010-08-23T15:51:05Z

‎Deploying new WARs

2010-08-23T15:49:59Z

@@ Line 60: / Line 60: @@
 To run the actual crawl do:
 <pre>
-$ ./bin/crawl_and_merge.sh
+$ ./bin/crawl-and-merge.sh
 </pre>

@@ Line 61: / Line 61: @@
 <pre>
 $ ./bin/crawl_and_merge.sh
 Finally, restart Tomcat (the Java app server) to make sure the new index is being used:

@@ Line 59: / Line 59: @@
 To run the actual crawl do:
 <pre>
-$ mv ./crawl-new-dir-merged ~/production-crawl
+$ ./bin/crawl_and_merge.sh
-And finally restart Tomcat (the Java app server) to make sure the new index is being used:
+Finally, restart Tomcat (the Java app server) to make sure the new index is being used:
 <pre>

@@ Line 141: / Line 141: @@
 To deploy a new war, do this:
-* sudo rm -rf /var/lib/tomcat6/search/ # clear the existing app to force redeployment
+* sudo rm -rf /var/lib/tomcat6/webapps/search/ # clear the existing app to force redeployment
 * sudo cp nutch-1.1.war /var/lib/tomcat6/webapps/search.war
 * sudo /etc/init.d/tomcat6 restart

@@ Line 141: / Line 141: @@
 To deploy a new war, do this:
 * sudo cp nutch-1.1.war /var/lib/tomcat6/webapps/search.war
 * sudo /etc/init.d/tomcat6 restart

@@ Line 66: / Line 66: @@
 This will read the seed files and run the crawl.  The result of this is a new index in the oenutch directory; the directories have a timestamp derived directory name.  For example, crawl-20090730201000 for a crawl run on July 30, 2009 @ 8:10 PM.  After the crawl completes you need to merge the new index with the old one.
-The production index lives in '''/var/www/discovered.labs.creativecommons.org/production-crawl'''.
+The production index lives in '''$HOME/production-crawl''' ($HOME is set to /var/www/discovered.labs.creativecommons.org).
 To merge the index run:
 <pre>
-$ ./bin/merge ./crawl-<timestamp>-merged /var/www/discovered.labs.creativecommons.org/production-crawl ./crawl-<timestamp>
+$ ./bin/merge ./crawl-<timestamp>-merged $HOME/production-crawl ./crawl-<timestamp>
 </pre>
@@ Line 99: / Line 99: @@
 == Managing curators and feeds ==
-On a6, in the /var/www/discovered.creativecommons.org/oenutch directory, running ./bin/feeds with no parameters shows the list of subcommands:
+On discovered.labs.creativecommons.org in the $HOME/code directory, running ./bin/feeds with no parameters shows the list of subcommands:
 <pre>

← Older revision		Revision as of 19:17, 30 August 2010
Line 143:		Line 143:
	* sudo cp nutch-1.1.war /var/lib/tomcat6/webapps/search.war		* sudo cp nutch-1.1.war /var/lib/tomcat6/webapps/search.war
	* sudo /etc/init.d/tomcat6 restart		* sudo /etc/init.d/tomcat6 restart
		+
		+	== Things the server administrator should know ==
		+
		+	=== JAVA_HOME ===
		+
		+	Many of our scripts require a JAVA_HOME environment variable to be set. For our convenience, we configured ''discovered.labs.creativecommons.org'' to have JAVA_HOME set for every user. We did that by adding this to '''/etc/profile''':
		+
		+	JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ ; export JAVA_HOME
		+
		+	=== Maximum open files ===
		+
		+	Tomcat and Nutch sometimes have problems opening files. This is because they've exceeded the number of open files that a process can have.
		+
		+	To address this, we added this to '''/etc/security/limits.conf''':
		+
		+	### For Tomcat etc.
		+	* soft nofile 4096
		+	* hard nofile 4096

@@ Line 94: / Line 94: @@
 <pre>
-$ sudo /etc/init.d/tomcat5.5 restart
+$ sudo /etc/init.d/tomcat6 restart
 </pre>

← Older revision		Revision as of 15:51, 23 August 2010
Line 140:		Line 140:

	To deploy a new war, do this:		To deploy a new war, do this:
		+
		+	* sudo cp nutch-1.1.war /var/lib/tomcat6/webapps/search.war
		+	* sudo /etc/init.d/tomcat6 restart

@@ Line 19: / Line 19: @@
 <pre>
 $ sudo su - discovered
 </pre>
@@ Line 135: / Line 136: @@
 setcurator [feed_url] [curator_url]
 </pre>