Creative Commons - User contributions [en]

Cchost/Developers/Victor's Dev Setup

2009-04-27T10:19:59Z

Michaelkrnac: /* Configuring Apache */

[[Category:ccHost]]
[[Category:ccMixter]]
[[Category:Developer]]
[[Category:Software]]
[[Category:Technology]]
{{cchost_head}}
{{lowercase}}

I recently had the opportunity to install a new development machine using the latest tools. This is how my ccMixter/ccHost development machine was set up.

==Version Info==

*Ubuntu: 8.04 - the Hardy Heron - released in April 2008.
*Apache: Apache/2.2.8 (Ubuntu)
*PHP: PHP/5.2.4-2ubuntu5.3 with Suhosin-Patch
*mySQL: 5.0.51a (Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2)
*Subversion: Client 1.4.6
*RapidSVN 0.9.4
*Meld: 1.1.5.1
*ccHost 5.0.1 (Committed revision 11342 Nov. 29, 2008)

==Ubuntu==
I used the [http://wubi-installer.org/ Wubi installer] to get Ubuntu on my Windows laptop.

I end up as user 'victor' in group 'victor'.

After installation, there were 195 'updates' and I got all of those.

==LAMP==
I then went out and got and verified the pieces of LAMP:

sudo apt-get install apache2
sudo apt-get install mysql
sudo apt-get install php5 libapache2-mod-php5
sudo apt-get install php5-cli

===Configuring Apache===
Because I'm using the PHP module (not CGI) Apache will perform operations in 'nix as a user in a group that will need write permissions to several directories. In order to ease this problem I change the default group that apache will run in to the same as me ('victor'):

cd /etc/apache2
sudo vi envvars

Then edit the line:

export APACHE_RUN_GROUP=www-data

and change '''www-data''' to '''victor'''

I next want to enable mod_rewrite:

cd /etc/apache2/mods-enabled
ln -s ../mods-available/rewrite.load rewrite.load

And you need to enable .htaccess overwrites:

cd /etc/apache2/sites-available
sudo vi default

Under ''<nowiki><Directory /var/www/></nowiki>'' change the line:

AllowOverride None

to:

AllowOverride All

I prefer to keep my localhost website for junky experiments so I create a virtual Apache host for ccMixter. Create the new file:

sudo vi /etc/apache2/sites-available/cchost

Enter the following:

<nowiki><VirtualHost *></nowiki>
DocumentRoot <nowiki>/var/www/cchost</nowiki>
ServerName cchost
<nowiki></VirtualHost></nowiki>

Then enable it:

ln -s /etc/apache2/sites-available/cchost /etc/apache2/sites-enabled/cchost

Then map the name for my local machine:

sudo vi /etc/hosts

Add the line:

127.0.2.1 cchost

===Setting up the ccHost Database===
Even though I'm going to mirror the ccMixter database here, it doesn't matter what the names are here locally:

mysql -p -u root
mysql> CREATE DATABASE ccmixter;
mysql> GRANT ALL ON ccmixter.* TO 'ccm_user'@'localhost' IDENTIFIED BY 'ccm_password';

===getID3===
I like '''curl'''

sudo apt-get install curl

I want to make the mirroring with ccMixter smooth so I create a directory that matches where we have getID3 on the ccMixter server:

sudo mkdir /var/getid3

Then I grab the zip from the net, unzip and do some clean up:

curl <nowiki>http://internap.dl.sourceforge.net/sourceforge/getid3/getid3-1.7.8b3.zip</nowiki> > getid3.zip
unzip getid3.zip
rm getid3.zip
rm -f -r demos

==Subversion==
The Creative Commons code server uses Subversion. There's a very usable and worthwhile GUI on Ubuntu and a fine diff program that I get:

sudo apt-get install subversion
sudo aptitude install rapidsvn
sudo aptitude install meld

===Writable Repository Access===
Since I need writable access to the CC code repository, I had to register my public key with the system.

If you only plan to do read-only updates from SVN you can skip this section entirely.

If you already have keys registered in your system you can skip this section entirely.

If you don't have ssh keys then you can use ssh-keygen and let the CC folks know about it to get writable access. And you can skip the rest of this section entirely.

My key was originally generated on Windows using the Putty tools. The 'nix system prefers OpenSSH version of the key files so I had to translate the PPK (Putty) version to that. I probably could have gone back to Windows to translate these but instead I installed the 'nix version the Putty tools:

sudo apt-get install putty

From my home directory I ran my key file through that to generate the OpenSSH key files:

puttygen -L victors_key.ppk > .ssh/id_dsa.pub
puttygen victors_key.ppk -O private-openssh -o .ssh/id_dsa

The .ssh/id_dsa naming convention is magic and required, the files must be private read-only:

chmod 600 .ssh/id_dsa
chmod 600 .ssh/id_dsa.pub

The '''ssh-agent''' daemon was already running in my install so all I had to do was register the keys (the passphrase was prompted for):

ssh-add

From this point on, all SVN access to the CC server would be done passing these keys back and forth and I never had to think about it again.

==Getting the Code==
Moving to /var/www I got the code:

svn co svn+ssh://svn@code.creativecommons.org/svnroot/cchost/trunk cchost

If you're only doing read-only updates then its:

svn co <nowiki>http://code.creativecommons.org/svnroot/cchost/trunk</nowiki> cchost

''IS THAT RIGHT????''

We need 'cchost' directory writable by the web server. The group is already correct ('victor') because I created it so all we have to do is enable write by the group:

chmod 775 cchost

For cchost log files we write them to a directory that is not accessible to the web. Just to make the mirroring easier I created a log directory in the same location as ccMixter's on my local machine. (The actual location is different than the one here but the steps are the same):

sudo mkdir /var/log/cchost
sudo chgrp victor /var/log/cchost
sudo chmod 775 /var/log/cchost

==Verifying ccHost Install==
Finally ready to see if our setup is alive. We start by restarting Apache to get our config changes to take hold:

sudo /etc/init.d/apache2 restart

Kill all instances of the browser, then browse to ''<nowiki>http://localhost/cchost</nowiki>''

NOTE: You may have to log out of Ubuntu or possibly reboot and get all the config stuff to take.

That brings up the 'cchost has not been properly install...' with a link to ''ccamdin' subdirectory. Follow that link to start the install.

After the install the '''Setting up your PHP environment''' screen tells me that ''/etc/php5/apache2/php.ini'' should be edited with different defaults and I go do that.

Click on the 'One more thing' link. Then rename the /var/www/cchost/ccadmin to something bogus.

Click through all the links and ccHost is up and running with defaults.

==Import ccMixter Config==

I have a bash script that will get the latest config from ccmixter.org and import it into the dev machine. The actual script using tons of piping in just three commands:

sqlite3 ~/.mozilla/firefox/vkuuxfit.default/cookies.sqlite "SELECT ':1',lastAccessed,':2',value FROM moz_cookies where name = 'lepsog3' and host = 'ccmixter.org'" | sed 's/:1|/ccmixter.org FALSE \/ FALSE /; s/|:2|/ lepsog3 /' > ccm_cookie
curl -b ccm_cookie <nowiki>http://ccmixter.org/export</nowiki> | sed 's/ccmixter.org/cchost.org/; s/Download, Sample, Cut-up, Share./ccHost DEV SITE/' > cch_import
cd cchost/bin
php -f cc-host-config-import.php ../cch_import

Below I break it up to annotate it.

The first thing I want to do is call <nowiki>http://ccmixter.org/export</nowiki> while logged in as admin. I can fake this by grabbing the ccMixter cookie out of Mozilla's Firefox. The cookie is usually in a directory like:

cd ~/.mozilla/firefox/vkuuxfit.default

I run a SQL query on it with place holders to a file called 'x':

sqlite3 cookies.sqlite \
"SELECT ':1',lastAccessed,':2',value FROM moz_cookies where name='lepsog3' and host='ccmixter.org'" \
> /var/www/x

I then substitute the place holders so that it forms a proper browser cookie:

cd /var/www
cat x | sed 's/:1|/ccmixter.org'''TAB'''FALSE'''TAB'''\/'''TAB'''FALSE'''TAB'''/; s/|:2|/'''TAB'''lepsog3'''TAB/'''' > ccm_cookie

NOTE: In order to make things clear in this wiki, I've bolded the tab characters - you should replace those with actual tab characters (ascii 7).

I now have a cookie and can use curl to call over to ccMixter and get the config as if I was logged in as admin. I save it to a file called 'xx'

curl -b ccm_cookie <nowiki>http://ccmixter.org/export</nowiki> > xx

The file 'xx' refers to ccmixter.org all over the place so I replace it with my dev domain. I also replace the site description in the banner so I don't forget which site I'm on (!):

cat xx | sed 's/ccmixter.org/cchost.org/; s/Download, Sample, Cut-up, Share./ccHost DEV SITE/' > cch_import

At this point I could browse to my dev site and using the import command:

<nowiki>http://cchost.org/import</nowiki>?i=../cch_import

but that's overkill considering we're right on the machine and can call a script that does the import:

cd cchost/bin
php -f cc-host-config-import.php ../cch_import

==Import ccMixter Data==

To mirror the data I use mysqldump on the CC server, sftp the results down to my dev laptop, then import the sql file using mysql command line. I leave that part as an exercise for the reader. (Just remember to NOT transfer the cc_tbl_config table.)

Cchost/Developers/Victor's Dev Setup

2009-04-24T13:55:59Z

Michaelkrnac: /* Configuring Apache */

[[Category:ccHost]]
[[Category:ccMixter]]
[[Category:Developer]]
[[Category:Software]]
[[Category:Technology]]
{{cchost_head}}
{{lowercase}}

I recently had the opportunity to install a new development machine using the latest tools. This is how my ccMixter/ccHost development machine was set up.

==Version Info==

*Ubuntu: 8.04 - the Hardy Heron - released in April 2008.
*Apache: Apache/2.2.8 (Ubuntu)
*PHP: PHP/5.2.4-2ubuntu5.3 with Suhosin-Patch
*mySQL: 5.0.51a (Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2)
*Subversion: Client 1.4.6
*RapidSVN 0.9.4
*Meld: 1.1.5.1
*ccHost 5.0.1 (Committed revision 11342 Nov. 29, 2008)

==Ubuntu==
I used the [http://wubi-installer.org/ Wubi installer] to get Ubuntu on my Windows laptop.

I end up as user 'victor' in group 'victor'.

After installation, there were 195 'updates' and I got all of those.

==LAMP==
I then went out and got and verified the pieces of LAMP:

sudo apt-get install apache2
sudo apt-get install mysql
sudo apt-get install php5 libapache2-mod-php5
sudo apt-get install php5-cli

===Configuring Apache===
Because I'm using the PHP module (not CGI) Apache will perform operations in 'nix as a user in a group that will need write permissions to several directories. In order to ease this problem I change the default group that apache will run in to the same as me ('victor'):

cd /etc/apache2
sudo vi envvars

Then edit the line:

export APACHE_RUN_GROUP=www-data

and change '''www-data''' to '''victor'''

I next want to enable mod_rewrite:

cd /etc/apache2/mods-enabled
ln -s ../mods-available/rewrite.load rewrite.load

And you need to enable .htaccess overwrites:

cd /etc/apache2/sites-available
sudo vi default

Under ''<nowiki><Directory /var/www/></nowiki>'' change the line:

AllowOverride None

to:

AllowOverride All

I prefer to keep my localhost website for junky experiments so I create a virtual Apache host for ccMixter. Create the new file:

sudo vi /etc/apache2/sites-available/cchost

Enter the following:

<nowiki><VirtualHost *></nowiki>
DocumentRoot <nowiki>/var/www/cchost<nowiki>
ServerName cchost
<nowiki></VirtualHost></nowiki>

Then enable it:

ln -s /etc/apache2/sites-enabled/cchost /etc/apache2/sites-available/cchost

Then map the name for my local machine:

sudo vi /etc/hosts

Add the line:

127.0.2.1 cchost

===Setting up the ccHost Database===
Even though I'm going to mirror the ccMixter database here, it doesn't matter what the names are here locally:

mysql -p -u root
mysql> CREATE DATABASE ccmixter;
mysql> GRANT ALL ON ccmixter.* TO 'ccm_user'@'localhost' IDENTIFIED BY 'ccm_password';

===getID3===
I like '''curl'''

sudo apt-get install curl

I want to make the mirroring with ccMixter smooth so I create a directory that matches where we have getID3 on the ccMixter server:

sudo mkdir /var/getid3

Then I grab the zip from the net, unzip and do some clean up:

curl <nowiki>http://internap.dl.sourceforge.net/sourceforge/getid3/getid3-1.7.8b3.zip</nowiki> > getid3.zip
unzip getid3.zip
rm getid3.zip
rm -f -r demos

==Subversion==
The Creative Commons code server uses Subversion. There's a very usable and worthwhile GUI on Ubuntu and a fine diff program that I get:

sudo apt-get install subversion
sudo aptitude install rapidsvn
sudo aptitude install meld

===Writable Repository Access===
Since I need writable access to the CC code repository, I had to register my public key with the system.

If you only plan to do read-only updates from SVN you can skip this section entirely.

If you already have keys registered in your system you can skip this section entirely.

If you don't have ssh keys then you can use ssh-keygen and let the CC folks know about it to get writable access. And you can skip the rest of this section entirely.

My key was originally generated on Windows using the Putty tools. The 'nix system prefers OpenSSH version of the key files so I had to translate the PPK (Putty) version to that. I probably could have gone back to Windows to translate these but instead I installed the 'nix version the Putty tools:

sudo apt-get install putty

From my home directory I ran my key file through that to generate the OpenSSH key files:

puttygen -L victors_key.ppk > .ssh/id_dsa.pub
puttygen victors_key.ppk -O private-openssh -o .ssh/id_dsa

The .ssh/id_dsa naming convention is magic and required, the files must be private read-only:

chmod 600 .ssh/id_dsa
chmod 600 .ssh/id_dsa.pub

The '''ssh-agent''' daemon was already running in my install so all I had to do was register the keys (the passphrase was prompted for):

ssh-add

From this point on, all SVN access to the CC server would be done passing these keys back and forth and I never had to think about it again.

==Getting the Code==
Moving to /var/www I got the code:

svn co svn+ssh://svn@code.creativecommons.org/svnroot/cchost/trunk cchost

If you're only doing read-only updates then its:

svn co <nowiki>http://code.creativecommons.org/svnroot/cchost/trunk</nowiki> cchost

''IS THAT RIGHT????''

We need 'cchost' directory writable by the web server. The group is already correct ('victor') because I created it so all we have to do is enable write by the group:

chmod 775 cchost

For cchost log files we write them to a directory that is not accessible to the web. Just to make the mirroring easier I created a log directory in the same location as ccMixter's on my local machine. (The actual location is different than the one here but the steps are the same):

sudo mkdir /var/log/cchost
sudo chgrp victor /var/log/cchost
sudo chmod 775 /var/log/cchost

==Verifying ccHost Install==
Finally ready to see if our setup is alive. We start by restarting Apache to get our config changes to take hold:

sudo /etc/init.d/apache2 restart

Kill all instances of the browser, then browse to ''<nowiki>http://localhost/cchost</nowiki>''

NOTE: You may have to log out of Ubuntu or possibly reboot and get all the config stuff to take.

That brings up the 'cchost has not been properly install...' with a link to ''ccamdin' subdirectory. Follow that link to start the install.

After the install the '''Setting up your PHP environment''' screen tells me that ''/etc/php5/apache2/php.ini'' should be edited with different defaults and I go do that.

Click on the 'One more thing' link. Then rename the /var/www/cchost/ccadmin to something bogus.

Click through all the links and ccHost is up and running with defaults.

==Import ccMixter Config==

I have a bash script that will get the latest config from ccmixter.org and import it into the dev machine. The actual script using tons of piping in just three commands:

sqlite3 ~/.mozilla/firefox/vkuuxfit.default/cookies.sqlite "SELECT ':1',lastAccessed,':2',value FROM moz_cookies where name = 'lepsog3' and host = 'ccmixter.org'" | sed 's/:1|/ccmixter.org FALSE \/ FALSE /; s/|:2|/ lepsog3 /' > ccm_cookie
curl -b ccm_cookie <nowiki>http://ccmixter.org/export</nowiki> | sed 's/ccmixter.org/cchost.org/; s/Download, Sample, Cut-up, Share./ccHost DEV SITE/' > cch_import
cd cchost/bin
php -f cc-host-config-import.php ../cch_import

Below I break it up to annotate it.

The first thing I want to do is call <nowiki>http://ccmixter.org/export</nowiki> while logged in as admin. I can fake this by grabbing the ccMixter cookie out of Mozilla's Firefox. The cookie is usually in a directory like:

cd ~/.mozilla/firefox/vkuuxfit.default

I run a SQL query on it with place holders to a file called 'x':

sqlite3 cookies.sqlite \
"SELECT ':1',lastAccessed,':2',value FROM moz_cookies where name='lepsog3' and host='ccmixter.org'" \
> /var/www/x

I then substitute the place holders so that it forms a proper browser cookie:

cd /var/www
cat x | sed 's/:1|/ccmixter.org'''TAB'''FALSE'''TAB'''\/'''TAB'''FALSE'''TAB'''/; s/|:2|/'''TAB'''lepsog3'''TAB/'''' > ccm_cookie

NOTE: In order to make things clear in this wiki, I've bolded the tab characters - you should replace those with actual tab characters (ascii 7).

I now have a cookie and can use curl to call over to ccMixter and get the config as if I was logged in as admin. I save it to a file called 'xx'

curl -b ccm_cookie <nowiki>http://ccmixter.org/export</nowiki> > xx

The file 'xx' refers to ccmixter.org all over the place so I replace it with my dev domain. I also replace the site description in the banner so I don't forget which site I'm on (!):

cat xx | sed 's/ccmixter.org/cchost.org/; s/Download, Sample, Cut-up, Share./ccHost DEV SITE/' > cch_import

At this point I could browse to my dev site and using the import command:

<nowiki>http://cchost.org/import</nowiki>?i=../cch_import

but that's overkill considering we're right on the machine and can call a script that does the import:

cd cchost/bin
php -f cc-host-config-import.php ../cch_import

==Import ccMixter Data==

To mirror the data I use mysqldump on the CC server, sftp the results down to my dev laptop, then import the sql file using mysql command line. I leave that part as an exercise for the reader. (Just remember to NOT transfer the cc_tbl_config table.)