MySQL export/import and charset problems

Today I discovered I messed up a database upgrade in MySQL. I had a rather large database which I exported from my main server and imported onto my testing server. Both machines had the same version of MySQL. I ran some update scripts on the testing server DB, verified the data, then exported the updated database, and reimported it into the main servers MySQL database.

Then I notice only after a few days that the data I re-imported had all the accented characters broken. Somewhere in my conversion the Latin1 charset did not export in UTF8 from Mysql as it was supposed to in MySQL 4.1.12 or greater!

The big issue here is that I could not simply run the upgrade again, as I would lose all changes to the database in the past week.

After some research I found the following bit of information using iconv ( iconv - Convert encoding of given files from one encoding to another)

Step 1
Re-export the data from the database normalling using
mysqldump—add-drop-table databasename > mydump.sql

Step 2
Use iconv to conver tthe format from UTF8 to ISO-8859-1
iconv -f UTF-8 -t ISO-8859-1 mydump.sql -o newdump.sql

Step 3
Re-import newdump.sql into MySQL with
mysql databasename

< newdump.sql

Now, if everything is normal you should have a great working database again. Or, if like with me, Murphy didn’t allow it to work normally, you might have gotten this error:
iconv: illegal input sequence at position 49823

It took me awhile to find the solution, but after much head banging, I finally found the following solution, whiched worked the charm, I might add!

Step 2
iconv -f UTF-8 -t ISO-8859-1//TRANSLIT mydump.sql -o newdump.sql

This will get the conversion done using transliteration, which means that when a character cannot be represented in the target character set, it can be approximated through one or several similarly looking characters (source: www.gnu.org).

Limiting sendmails Max Recipients

There have been a lot of attempts lately on peoples servers to send spam through online forms. This is done by creating a crafty email with a carefully placed Bcc: tag which is then remotely submitted to your unsuspecting form on your server.

All the email that goes through your form now from this person script will look like it came from your server. And hey, sendmail won’t know what is going on because it thinks you, a trusty ole soul, is sending this bulk email.

Apart from making your forms secure, you can deter the spammers by limiting how many recipients sendmail will send to with each email. Currently, if a spammer puts 100 email address in the fake Bcc: field in the spam, sendmail will gladly forward that email to those 100 recipients. All within the matter of a few minutes.

To limit this amount simply follow these instruction:

Step 1 - Backup your working copy of sendmail.cf before starting.

cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.working

Step 2 - Modify the existing copy of sendmail.cf

pico -w /etc/mail/sendmail.cf

Find the following line:

#O MaxRecipientsPerMessage=0

and change it to look like this:

O MaxRecipientsPerMessage=15

Save your file, and restart sendmail:

/sbin/service sendmail restart

Sendmail will now only send an email that has 15 or less recipients. Either in the To:, Cc: or Bcc: fields.

Happy mailing!

Creating a new repository in Subversion

For no other reason than ‘I can never remember’, this entry is being created. Most of my projects are ongoing, and rarely do I find the need to create new repositories. So when the time comes to run svnadmin create, I often find myself pondering the correct logistics of the svn import commmand.

This entry will assume you have a working subversion on your server. If you are on Ensim and you do NOT have subversion working on your server, then look no further than this article I wrote.

Now, let’s get going:

1 - Create the tree for the repository:
cd /tmp
mkdir myproject
cd myproject
mkdir trunk
mkdir branches
mkdir tags
cd trunk
cp -R /path/to/my/project ./
cd ../../

2 - Create the new repository which, for the purposes of this entry, we will call ‘myproject’. I am using the same naming convention that I used when I set up subversion. The path to my repos is where my subversion repos are officially kept according the apache configs I set up in this article.

svnadmin create /home/virtual/domainname/home/svn/repos/myproject

3 - Import the repos:

svn import /tmp/myproject file:///home/virtual/domainname/home/svn/repos/myproject -m “initial import”

You should see the following zip by on your screen:

Adding /tmp/project/branches
Adding /tmp/project/tags
Adding /tmp/project/trunk
Adding /tmp/project/trunk/.... your files here
.
.
.
#

Assuming it all went well, don’t forget to modify your svn-auth-file and svn-access-file.

Replace DB content on the fly

This appears to be a little known fact, so it has earned the right to be posted as a how-to on my site.

If you have a database with lots of content and you discover to your dismay that you have made a mistake on some of the text, in many rows, you might find yourself asking “How am I going to fix all those errors?”

For example, you have a table with 5000 rows and one column is a URI. You have entered the URI in this database as http://www.mystartupcompany.com/index.php?id=4001, http://www.mystartupcompany.com/index.php?id=4006, http://www.mystartupcompany.com/index.php?id=4009 and so on.

Now you discover that you can have the URI’s in a search engine friendly format that looks like this: http://www.mystartupcompany.com/product/4001, http://www.mystartupcompany.com/product/4006, http://www.mystartupcompany.com/product/4009 and so on.

Can you imagine having to go through the database one by one and correct all 5000 URI’s?

Relax, there is a simple and fast way to do it.

We will assume for arguments sake, that the table name is products, and the column name to modify is prod_uri.

Simply run this command:
UPDATE products SET prod_uri=REPLACE(prod_uri,‘mystartupcompany.com/index.php?id=’,‘mystartupcompany.com/product/’);

Note that the above code is all on one line.

Within a second or two, all the columns and rows will be updated.

Many thanks to Rob over at MacOSXHints for bringing this to my attention.

Page 18 of 18 pages ‹ First  < 16 17 18