ameir dot net

Using rsync with CIFS or SMB (Windows) Destination

This took me a good bit of reading and trial/error, but here are rsync options that I found to work well with a Windows target (tested with a CIFS mount) from a Linux ext3 filesystem without errors.

rsync -rvztO --progress --stats --copy-links --dry-run /source/path /destination/path

Bing Site Search in PHP with Pagination

This is a simple yet powerful way to get Bing's Site Search API working on your site in PHP. Simply update the $config variables below and you'll be good to go.

<?php
define('TITLE', 'Search Results');
include('header.php');

$config['appid'] = "01234567890"; // change this
$config['search_domains'] = array("domain1.com", "domain2.com");
$config['results_per_page'] = 20;

$url = "http://$_SERVER[HTTP_HOST]$_SERVER[PHP_SELF]";
?>

<?php echo (isset($_REQUEST['q']) ? "<h1>Search Results</h1>" : "<h1>Search</h1>"); ?>
<form method="get">
<div style="text-align:right">
<input type="text" id="q" name="q" value="<?php echo (isset($_REQUEST['q']) ? $_REQUEST['q'] : 'Search...'); ?>"/>
<input type="submit" value="Search" name="submit" id="searchButton" />
</div>
</form>

<?php
if (isset($_REQUEST ['q'])) {
$query = htmlentities($_REQUEST ['q']);
if (strlen($query) < 1)
echo "You didn't type anything!";
else {
if (isset($_REQUEST ['page'])) {
$page = htmlentities($_REQUEST ['page']);
$offset = (($page - 1) * $config['results_per_page']);
} else
$offset = 0;

$result = "http://api.search.live.net/json.aspx?Appid={$config['appid']}&sources=web";
$result .= "&query=" . urlencode("site:" . implode(" OR site:", $config['search_domains']) . " " . $query);
$result .= "&Web.Count={$config['results_per_page']}&Web.Offset=$offset";
//echo $result;
$result = file_get_contents($result);
$result = json_decode($result);
echo('<ul ID="resultList">');
foreach ($result->SearchResponse->Web->Results as $value) {
echo('<li class="resultlistitem"><a href="' . $value->Url . '">');
echo('<h3>' . $value->Title . '</h3></a>');
echo('<p>' . $value->Description . '</p>');
}
echo("</ul>");

// echo("<pre>");
// print_r($result);
// echo("</pre>");
$results = $result->SearchResponse->Web->Total > 1000 ? 1000 : $result->SearchResponse->Web->Total;
if ($results > 0)
echo $results . " results <br />\n";
else
echo "Sorry, no results to display. <br />\n";

$pages = (int) ($results / $config['results_per_page']);
// echo "$pages pages <br />\n";
// pagination
// echo "Page #s:<br />";
for ($i = 1; $i <= $pages; $i++) {
echo "<a href=$url?q=$query&page=$i>$i</a> ";
}
}
}
?>

<?php include('footer.php'); ?>

MySQL Backup to FTP and Email Shell Script for Cron v2.2

It's been a while since I've publicly made updates to this script, but I did make some tweaks over the years that I'd like to share.

Here are some changes over the last version:

Delete old backups via FTP
Backup to multiple FTP servers
More efficient backups
Add time in filename (allows for multiple backups/day)
More verbose/better error detection

With that said, follow the link below to download the new version:
mysqlbackup-2.3.sh

You can view the previous version here.
If there's a feature that's missing that you'd like to see, leave a comment below.

PHP Function to Return Lowest MX Record

This is a quick and simple one, but I admittedly started off with a for loop before coming across array_multisort. This turned out to be very useful for an application I'm working on.

// example
echo "Google's lowest MX is: " . lowestmx("google.com");

function lowestmx($server) {
getmxrr($server, &$mxhosts, &$weight);
array_multisort($weight, $mxhosts);
return $mxhosts[0];
}

Yahoo! Mail IMAP Proxy

UPDATE: It was fun, but this proxy is no longer needed. You can connect directly to Yahoo! via IMAP with the following servers:

imap.mail.yahoo.com port 993 (SSL)
smtp.mail.yahoo.com port 465 (SSL)

There are a few Yahoo! Mail IMAP proxies out there (YPOPS!, FreePOPs), but to be honest, I haven't had much luck with either of them. Additionally, switching webmail interfaces to Asia or Classic or whatever can be a bit of a hassle.

Yahoo! supports IMAP on some mobile devices, and now also allows it for anyone using the Zimbra mail client. Unfortunately, I'm not too fond of that client and much prefer my Thunderbird; too bad it isn't natively supported.

Luckily, Yahoo's IMAP implementation isn't too far off from what other clients recognize; it simply requires a "ID ("GUID" "1")" to be issued before logging in. No clients do that (except for these hacked up versions of Thunderbird), so my workaround was to create a simple IMAP proxy.

This proxy simply takes the commands sent to it by your mail client, passes them on to Yahoo, and relays them back to your client. It all the while looks for your client to issue a "login" command so that it can inject the "id" command to unlock IMAP access.

This program is written in C and has been tested on Linux and Windows (via Cygwin). Read the release notes in the source for more information. I haven't written in C for a while, and I know that this program can be improved. If anyone does so, I'd like to hear from you.

Connection settings:
Username: yahoo_username@yahoo.com
Hostname: localhost
Port: 3490
SSL: off

Download latest source

Download Windows Cygwin version

View Google Code Homepage

Installing Windows XP over a Network using PXE

I just spent countless hours trying to revive an old laptop with seemingly no hope left in it. Its CD-ROM drive is bad, it doesn't support booting from USB, and it has no floppy drive. After backing up the hard drive using a USB caddy, I tried countless ways of loading up bootdisks over PXE using MEMDISK and PXELINUX. I got very close several times, but unfortunately both FreeDOS and MS-DOS failed me. The closest I got was by partitioning the hard disk such that I had a 1GB FAT partition at the end with the XP installation files copied to it. Running winnt32.exe from DOS brought me a lot of hope, but also a lot of pain, namely the error "Setup is out of memory and cannot continue." I hacked away and the config.sys file and kept editing various bootdisks, but no luck in the end. I tried dozens of things, but I'll skip that all and get to the good stuff: how I got it working.

Continue reading "Installing Windows XP over a Network using PXE"

Getting Ubuntu 9.10 Karmic Koala working in OpenVZ

If you're an Ubuntu fan like me and use (or want to use) OpenVZ, you might not be too excited that the OpenVZ site doesn't yet have the latest edition of Ubuntu, 9.10 Karmic Koala, available as a precreated template. Luckily, you can find them here.

For my purposes, I downloaded ubuntu-9.10-minimal_9.10_amd64.tar.gz. After setting up a virtual instance with Proxmox and opening up a VNC session, I found that this template is way more minimal than I ever expected. In all reality, that isn't so bad. The lighter, the better. Unfortunately, since I was setting up this new instance as a server, I had a good bit of work ahead of me.

Firstly, networking wasn't working at all with my bridged setup out of the box. The solution:

edit /etc/network/interfaces and configure your NICs. I had to comment out some properties of the venet0 interface to get things working. Here's my final config file:

CODE:

auto lo
iface lo inet loopback
auto eth0
# iface eth0 inet dhcp
iface eth0 inet static
        address 10.x.x.x
        netmask 255.255.255.0
        gateway 10.x.x.y

auto venet0
iface venet0 inet static
        address 127.0.0.1
        netmask 255.255.255.255

Notice that I'm using a static IP address. You may want to try DHCP starting out since it'll make things easier. Confirm network connectivity by pinging a remote host. Don't forget to edit /etc/resolv.conf if you need to.

Now, I need to get some goodies installed on this thing. In the end, I'm looking to install Virtualmin and host some websites, so I need to prep for that. Luckily, Ubuntu makes it easy to get things started quickly. Start off by issuing:

apt-get update && apt-get install tasksel && tasksel

An ncurses dialog should appear. Go ahead and select what you need. I can tell you that I learned the very hard way by not initially selecting "Basic Ubuntu Server." If you don't select that (don't worry, it's only a couple of small packages), none of your services will start at boot-time and you'll have to start each of them manually.

From there, you should be on your way. I strongly recommend Webmin (add the apt repo) to ease server administration tasks.

Also, before I leave you be, if you're not using Proxmox to perform all of this OpenVZ magic, you're definitely missing out. Hurry up and convert already!

Setting up Virtualmin on Ubuntu 9.10 Karmic Koala

According to the Virtualmin website, the latest version of Ubuntu that is supported is 8.04 LTS. That's probably a safe stance to take since non-LTS versions of Ubuntu have a six-month lifecycle, and most serious hosting companies would stick with LTS versions. For those of us who live on the bleeding edge and want to try Virtualmin on the latest version of Ubuntu, 9.10 (Karmic Koala), you'll find that the standard method of installation (the install.sh script on their site) will fail.

Luckily, the solution is simple.

Add the following repository to your sources.list file:

deb http://software.virtualmin.com/gpl/debian virtualmin-universal main

Then run the following as root:

cd /root
wget http://software.virtualmin.com/lib/RPM-GPG-KEY-virtualmin
wget http://software.virtualmin.com/lib/RPM-GPG-KEY-webmin
apt-key add RPM-GPG-KEY-virtualmin
apt-key add RPM-GPG-KEY-webmin
apt-get update ; apt-get install webmin webmin-virtual-server

Webmin and Virtualmin will now be installed. Note that the repository may not be as up-to-date as the direct download link. If that scenario arises, simply find the URL of the latest Virtualmin package from the link, and do:

wget http://download.webmin.com/download/virtualmin/webmin-virtual-server_3.77.gpl_all.deb ; dpkg -i webmin-virtual-server_3.77.gpl_all.deb

Cheapest SSL Certificates

Interested in securing your website? There are several types of certificates out there at wildly varying prices, but they all provide exactly the same function--encryption. In most cases, you're no more secure with a $300 certificate than a $10 one; the difference lies in the verification the certificate company performs to confirm your identity. In short, if you're not a bank or think your users won't trust you using a cheapo certificate, then you'll be fine with a cheapo.

Below the cheapest SSL certificates I found online and have personally used:

~~AlphaSSL from Dynadot ($41.50/5 years)~~

Positive SSL from CheapSSLs.com ($24/3 years)

RapidSSL from CheapSSLs.com ($27/3 years)

RapidSSL from Servertastic ($50/5 years)

As you can see, you get discounts if you pay for multiple years in advance. If you plan on keeping the domain for a while, it shouldn't hurt--just be sure to keep the certificate and key in case you decide to switch servers over time.

If you know of any other deals, let me know!

Creating a customized OpenVPN installer - Round 2

In a previous article, I outlined the steps I took to "roll" my own customized OpenVPN installer, and it worked like a charm back then. OpenVPN has gone through several revisions since, and getting things running (especially on different architectures) with the new versions just doesn't work so well. Luckily, there's another way to approach this problem and have OpenVPN installed as it was intended (i.e. the correct TAP driver will be detected and the shortcuts where they belong). This method is more of a workaround, but definitely works. Note that with this method you can't rename OpenVPN to "MyVPN" or whatever like the previous method.

This method employs the use of an SFX that installs the vanilla OpenVPN installer as downloaded from their site, then automatically installs the keys afterwards. You have the option of making the install entirely silent as well--it's all up to you.

Prerequisites:

7-ZIP SFX Maker (version 2.0 at the time of writing)

7-Zip (or another 7-Zip-capable archiver such as IZArc)

OpenVPN (version 2.1_rc19 at the time of writing)

Your OpenVPN keys

Steps:

First, package your OpenVPN keys with 7-Zip. If you want multiple OpenVPN connections configured on the same machine, it's a good idea to have each connection's keys in a subfolder. Ensure that each connection has a .ovpn or .conf file with a unique name.

Next, open up 7-ZIP SFX Maker and add your archived keys by clicking on the "+" symbol. Now, go to the "General" tab. Under "Extract to specified folder", enter %ProgramFiles%\OpenVPN\config. You can right-click for some preset environment variables if you want to change the path.

You can change other options if you'd like. I have "Beginning extraction of keys..." in the "Begin prompt" field of the Text tab. From here, you're ready to click "Make SFX".

Now, create another 7-Zip archive with your new SFX and the OpenVPN installer. Add this archive to 7-ZIP SFX Maker. In "General" enter %tmp% or another writable directory in the "Extract to specified folder" field. I have "Allow user to change extraction path" checked as well--this all depends on your environment and needs. Under the Text tab, I have it filled as follows:

Under the Shortcuts tab, I created a shortcut to OpenVPN GUI in Startup so that it starts on login.

Now, (and this is important), go to the Execute tab. We will tell the SFX maker to run the OpenVPN installer then run the SFX for the keys. Be sure to list them in that order. Note that I have a /S in the first entry because I want OpenVPN to install silently. Sadly, not everyone at my company knows how to install software, and I'm satisfied with the setup defaults anyway.

You're now ready to "Make SFX"! Your new installer will be placed in the folder your files were in.

You might want to play around with settings until the installer suits your tastes.

Redirect HTTP traffic to HTTPS with IIS

There are many situations where you'd want a site to be accessible only securely, and there are several resources online on how to accomplish this. Unfortunately, I couldn't find a perfect solution for me. Influenced by this info and following the steps outlined here (under In Windows Server 2003 (IIS 6.0)), I came up with the following:

CODE:

<%
   If Request.ServerVariables("SERVER_PORT")=80 Then
      Dim strSecureURL

  strSecureURL = Replace(Request.QueryString,"http","https")  ' entire old URL, but w/https
  strSecureURL = Replace(strSecureURL,"403;","") ' remove "403;" from beginning
  strSecureURL = Replace(strSecureURL,":80","") ' remove port # that's appended to server name
  Response.Redirect strSecureURL
    End If
%>

It works well for my purposes, but of course, your mileage may vary.

More resources:

1. http://support.microsoft.com/kb/239875
2. http://support.microsoft.com/kb/839357
3. http://blog.opsan.com/archive/2005/04/17/395.aspx

GWMover: WAN Failover Script for Linux

This is a script I wrote a while back but never posted up. It allows you to switch from one gateway to another by repeatedly pinging your ISP gateway (or another external IP) and--upon detection of failure--switching to another gateway. If the primary line is up upon the next run, we switch over to it.

This isn't a very advanced script and doesn't factor in other considerations, such as the state of the physical interfaces. It simply attempts a number of pings, and considers the link dead or unreliable if they don't all come back.

If you have mutt installed, you will get an email alert when the primary link is considered 'dead.' You can easily change this behavior by modifying the script.

It will probably work on other POSIX/Unix-like OSes, but I've only tested it on Linux in a BASH environment. With all that said, here are the goods:

CODE:

#!/bin/bash
# GWMover:  WAN/Gateway failover
# Pings gateway and switches to backup line when it goes down.
# Reverts to primary line when gateway becomes accessible.
# Modified version of script found at:
# http://www.howtoforge.com/forums/showthread.php?p=55402#post55402
#
# Last modified:  August 2, 2007
# Ameir Abdeldayem

PRIMARYPUB="66.92.162.1"
PRIMARYGW="10.0.5.1"
SECONDARYGW="10.0.0.1"

EMAILS="email@ddress1 email@ddress2"

function is_host_alive() # Returns success or failure as boolean
{
PACKETS=3
TRIES=5
IP=$1 # saving contents in $1 before is used
echo "Pinging $IP"
i=0
while [ $i -lt $TRIES ]; do
REQUEST=$(ping -qc $PACKETS $IP |grep packets| cut -d" " -f1,4)
set -- $REQUEST
echo -e "Try $i: $1 $2     "
if [ $1 != $2 ]; then
echo "Bad news. $1 sent, $2 received."
# exit
exitcode=1
else
echo "Great! $1 sent, $2 received."
exitcode=0
fi
i=$(expr $i + 1)
done
return $exitcode
}

CURRENTGW=`route | grep "default" | awk '{print $2}'`
function changegateway()
{

if [ $1 != $CURRENTGW ]; then
echo ".  We need to change the default route. After: $1 Before: $2"
route add default gw $1
route del default gw $2
echo "The routing table has been changed!"
route # view current routing table
echo -e "$4 \n\nHere is the updated routing table:\n$(route)" | mutt -s "$3 ($(hostname))" $EMAILS
else
echo ", and we are already pointing to it."
fi

}

function test()
{
echo $1 $EMAILS
}

##################
# main starts here
##################

if is_host_alive $PRIMARYPUB ; then
echo -n "The primary gateway is alive"
SUBJECT="$PRIMARYPUB IS NOW UP"; BODY="The server $PRIMARYPUB is now up and was pinged via the current gateway of $CURRENTGW."
changegateway $PRIMARYGW $SECONDARYGW "$SUBJECT" "$BODY"
# test $PRIMARYGW
else
echo -n "The primary gateway is dead <img src="http://www.ameir.net/blog/templates/default/img/emoticons/sad.png" alt=":-(" style="display: inline; vertical-align: bottom;" class="emoticon" /> We should be on the backup line"
SUBJECT="$PRIMARYPUB IS DOWN"; BODY="The server $PRIMARYPUB could not be pinged via the current gateway of $CURRENTGW."
        changegateway $SECONDARYGW $PRIMARYGW "$SUBJECT" "$BODY"
fi

Download it here.

Sync SVN repository with remote FTP server

This script was written for the company I work for well over a year ago, but I just decided to release it to the public now. It has been in use all of this time and is still in use now, so that should tell you that it's a pretty reliable script. Any bugs we encountered over the course of using this program have been fixed. Hopefully you'll have the same luck as us.

Basics:

This script is run as a SVN hook (post-commit). After committing changes to your subversion repository, this script is invoked and synchronization begins. The script creates a local copy of the repository, and files just committed are sent via FTP.

Usage:

One copy of the script can be used for an unlimited number of repositories, so the script can be placed in a globally-available location. In our scenario, we placed the script at /usr/local/bin/svn2ftp.sh . There are four command-line arguments to the script that are required. The first two arguments can be kept the same as below, as they're passed to the script by subversion. The third parameter is the path between the repository and the trunk (e.g., if your SVN path is http://<host>/svn/repo/dir/trunk, then you'd put "dir" as the parameter; otherwise, you'd put "." if there's no directory in between). The fourth parameter is the full path to the config file of the repo. A sample to put in repo/hooks/post-commit:

CODE:

# Update this repo with remote FTP server
DATE=`date +'%m-%d-%Y'`
/usr/local/bin/svn2ftp.sh "$REPOS" "$REV" "." "/svn2ftp/conf/repo.conf" >> /svn2ftp/logs/repo-$DATE.log

We have our directory structure as follows, but you can change it as you please:

/svn2ftp/conf # Where config files are stored

/svn2ftp/logs # Where log files are stored

/svn2ftp/repos # Where raw repo contents are stored

A sample config file can be found below.

The code:

I initially wrote this program using the Linux 'ftp' program. It works well and all, but if I encounter server or any other errors, the program still exits with a successful error code. This was a bummer in our situation, so I wrote an FTP client in PHP that reports errors correctly. When an error is encountered, files that were not uploaded successfully will be retried upon the next commit. Links to both versions, as well as the config file, are below:

Version using Linux 'ftp' program

Version using PHP FTP program

Sample config file

PHP FTP Updater

Site back online

The site was down for a while (since August 21) due to network issues where the previous server was located. I found it easier to just move the site to another server than to investigate the issue (I'd have to work out a time to go to the 'colocation center' -- my friend's dorm room). I've been meaning to move the site to this server anyways, so I guess it all worked out, despite the mega downtime.

Sorry for the inconvenience, and welcome back .

Using OpenDNS on Your Fon Router

There is large interest in OpenDNS, and I happen to be a fan of it myself. Sadly, Fon routers, when connected directly to a modem and using DHCP, use the ISP's DNS servers and don't allow you to override those settings. Fortunately, there is a loophole in older Fon firmware that allows you to enable a BusyBox shell (so you can connect to it via SSH). This is actually a security hole and shouldn't be a good thing at all, but hey, I'm kind of glad it exists. Anyways, let's get started.

Enable SSH access to your Fon router by following these instructions. Be sure to enable permanent SSH access, as it'll be very useful later. When your Fonera updates to newer firmware that doesn't allow code injections, SSH will still be enabled. Be sure to leave your network cable unplugged until after you enable SSH permanently or you might form some gray hairs.

Now, in a shell, let's create a startup script that overrides the ISP's DNS settings after the network is initialized. Type the following:

vi /etc/init.d/S45opendns

Now, in the vi window, paste this (press i to insert text):

echo "nameserver 208.67.222.222" > /etc/resolv.conf
echo "nameserver 208.67.220.220" >> /etc/resolv.conf

This will "blank" /etc/resolv.conf and load it with the OpenDNS settings. Save and exit vi (press the esc key then type :x).

Now, to make the file executable type:

chmod +x /etc/init.d/S45opendns

You're all done! OpenDNS settings will be persistent across reboots. You can activate the settings now without having to reboot by simply running your new script:

/etc/init.d/S45opendns

If DHCP renewals cause your new nameserver settings to be overwritten to the ISP's, you can create a cron job that runs every few minutes (you can pick a time based on your ISP's DHCP lease time):

Type crontab -e in the terminal window. When vi opens, insert the following line at the end:

15 * * * * /etc/init.d/S45opendns

This will run your new script every 15 minutes, so there will never be more than a 15-minute period where you will not be using OpenDNS.

That should be all you need to get you going. If you're using more advanced features of OpenDNS, you can use DNS-O-Matic on your Fon to keep OpenDNS informed of IP changes. Comment below with your experiences.