Backing up and Reformatting Windows XP Computer

This is always such a hassle, your computer is at it's life's end, you've had the system for over 3 years and you've installed and uninstalled programs, games, and who-knows-what else over the past long while and it's time to backup and reformat to squeeze the last bits of performance out of that bad-boy comp you bought 4 years ago. You've upgraded the memory and video card so it's not so out of date and perhaps malicious infections, viruses, and/or spyware cause your computer to not live up to it's it's potential.

Time to backup your files to an external source and reinitialize XP or go to Windows 7. Here are some of the directories in XP you'll want to look for important files to backup:

C:\Documents and Settings\[UserName]\Desktop
C:\Documents and Settings\[UserName]\My Documents
C:\Program Files

As well as any files you might have saved directly under the C:\ (root) of your hard drive or subfolders in the root. Note: there are also hidden folders inside "Documents and Settings" such as local settings and application data that might contain saved games or other important files you'll want to backup.

To make sure I got all my files under C:\Documents and Settings while backing up I used robocopy to copy files off onto a separate hard disk I connected using this cool SATA/IDE to USB adapter I bought awhile back and my 500 gig Seagate HD. Here is the command I used to copy the files:

ROBOCOPY "C:\Documents and Settings" E:\Backup /COPYALL /B /MIR /R:0 /W:0 /NFL /NDL

This took a really long time, especially because I had some photos on my desktop that I hadn't moved before the backup, but at least I could have the peace of mind knowing all my files were transferred despite the failure of copying through windows explorer giving the file in use error.

You could also run Robocopy for "C:\Program Files" but I found it was easier just to do a spot-check on those folders since most modern programs save their user profile and customizable data files inside the user's Documents and Settings directory.

Before you make the plunge of reformatting to avoid the headache and pain of lost data, use disk analysis tools such as WinDirStat to make sure all the data you need to backup is archived off your main drive. If you need to make backups within installed programs such as Local Mail Systems (i.e. Thunderbird and Outlook), Databases (such as MSSql, MySql, Oracle, etc...) or Source Control Systems (i.e. CVS, SVN, VSS, etc...), perform those backups with the appropriate programs and make sure you have the files archived somewhere safe.

Once you're ready, go for it! Make sure you have your XP or Windows 7 disk and hardware driver disks handy as well as files for other programs you use regularly. If your computer boots up at the BIOS screen but does not prompt you to boot off the CD ROM for the O/S you may have to go into the CMOS screen to change the boot order to include the CD/DVD before the HD.

It is so nice to run a clean system right after a reformat, of course you'll want to install your Chipset drivers (which includes sound drivers if the motherboard supports it) as well as your video drivers and reinstall an anti-virus solution as soon as possible, I suggest AVG Free, it is a very functional virus scanner for only $0!!!

If everything goes smoothly you'll have a working system that runs much more efficiently than before, I'm still looking for a good anti-spyware program and had been using AdAware before but if anyone has found a better free spyware-killer let me know.

Update October 13th 2009:

I've decided to try Microsoft Security Essentials which serves as both my virus scanner and my anti malware and spyware solution, and it even works in XP!

To be fair, I hadn't performed a Full Scan within AVG before I uninstalled it on my fresh XP install but Microsoft Security Essentials says "Preliminary scan results show that malicious or potentially unwanted software might exist on your system." during it's full scan, so I patiently wait for it to finish so I can see what it has detected...

It found one: "Spyware:Win32/Aureate" item and also: "Backdoor:Win32/Trenk!rts" which it was able to clean out. Overall I'm pretty happy with the Microsoft Security Essentials application, the UI is nice and clean and since it is free and provides antivirus, and spyware/malware protection I know I'm pretty well covered.

Windows Time Service: W32Time - Synchronising time between servers

We have the need at work to Synchronize the clocks of our application servers with the database server which should be a relatively simple task and it's not too difficult given you have a domain controller in your network, however we don't want to require a domain controller for our software to run properly and getting the Windows Time Service to sync the clocks between servers without a Domain controller was a bit of a headache.

In this article: How to configure an authoritative time server in Windows Server Microsoft describes the different registry entries used in Windows Server 2003 to configure the W32Time service, if you also read the article: Configure a manual time source for a selected client computer you will become more familiar with the commands for querying and synchronizing against a remote time server.

First I tried following the directions under: "Configuring the Windows Time service to use an internal hardware clock" because I didn't need the server to sync to an external time source, I just wanted to sync the clocks of the client machines to my database machine.

After configuring the server I issue the following command from the client machine:
w32tm /config /manualpeerlist:peers /syncfromflags:manual /update

Where peers is the server's name.

However as soon as I issued the command:

w32tm /stripchart /computer:target /samples:n /dataonly

from the client machine, the command errored out with: error: 0x800705B4
Or if I give the resync order to the client machine using the command:
w32tm /resync
it replies back with:
"The computer did not resync because no time data was available."


After searching on Google and not turning up anything very helpful I went ahead and followed the instructions under the "Configuring Windows Time service to use an external time source" section. For the registry entry that dictates which server to use as the external time source (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters), I used the value: time.windows.com,0x1

Using this setup I was able to get my database server to sync with time.windows.com and my web servers to sync with the database server (however at one point I had my db server synced up with the Domain Controller (DC) and it was working that way as well so I'm not sure if performing that step made it work even after changing the configuration to externally sync up with time.windows.com). As I'm writing this blog post I'm retracing my steps on another set of machines and I can't get it to work so maybe there is some sort of problem with having IIS on your time server because I found other posts related to that.

Concluding, time syncing in windows is a real pain (to say the least) and it seems that getting it to work in an environment without a Domain Controller is even more difficult. I'll update this post as I find new things or if we need to setup another cluster I may learn more. Hopefully this helps someone, if you have more experience with this and know what the magical compination of steps and registry entries are to make it work let me know I'd love to hear. I'm afraid that if I have to figure this out again on another cluster I'm going to be in real trouble.

DupeMan - Duplicate File Management Program Updated

It's been about 3 years since I've updated and released changes to my original DupeMan Duplicate file management tool, but I've been able to make changes to it in my spare time and when I get bored, I upgraded to .NET 2.0 framework and added the ability to save and load the results of a check for duplicates indexing process as well as a new MP3 Playlist dockable control.

Download DupeMan Here.



Feel free to pass this program or link on to your friends or anyone you think might be able to use it (those who's collection is more quantity than quality).

If you do install it and try it out, let me know any feedback you have,
-Matt Palmerlee

(Note: you need to have at least .NET 2.0 installed on your system to work.)

Configuring Google Apps to host your Domain for free

I've recently setup a new domain name for a new S/W Development Consulting company I'm creating called Mastered Software (MasteredSoftware.com). I was planning on creating my new website using C# .NET and Visual Studio .NET 2005, because that is the platform I'm most comfortable with, however, finding affordable Windows hosting that supports the .NET framework is not easy, the cheapest I could find was $10 a month if you bought 2 years at a time, which isn't much compared to other hosting solutions, but for a small business even this adds up quickly.

I was also recently checking out Google's App Engine and Google Apps for your domain services, and what's cool is that you can host your domain and even email through the google apps for your domain service! The best part is, for small companies that don't need lots of storage and features it's free!

Here are the steps I took to have my domain hosted by google:

1. Start by signing up for Google Apps for your Domain at: http://www.google.com/a/
2. You will be prompted to verify ownership of your domain during setup, there are a couple of options to do the verification, including adding a special CNAME record in your domain name registrar's system to point to google.com, this is the method I used for verification because it was the most straightforward. While you are there, you may want to add additional CNAME records for your domain which point to ghs.google.com (i.e. masteredsoftware.com and www.masteredsoftware.com).
   
3. Enable the applications you want such as email, calendar, etc... For creating the content for my main site I choose to use the Google Pages application because it's very easy, configure the web address for google pages to point to www. for my site it is www.masteredsoftware.com click on the "Change URL" link from the Web Page settings screen to update the web address. Create a basic home page for your site through google pages so you have something to look at when you go to your domain, don't forget to publish your changes so they are live on the site.
   
4. At this point you should be able to browse to the "www" version of your site (if you setup the CNAME records in step2 for www.yourdomain.com to point to ghs.google.com).
5. Unfortunately there is no way at the moment to have the "naked" domain version of your site point to your google pages site ,unless perhaps you have your domain hosted through google domain hosting, I don't know because I haven't tried that service. You maybe able to setup a forward or redirection from the naked domain to the www version in your hosting provider's DNS settings, with my hosting provider I was not able to do that and have some more of the advanced features for my domain such as redirecting the MX records to google. Fortunately you can host your naked domain using the google apps framework.
6. Once you have signed up for the google apps framework, add your naked domain (in my case masteredsoftware.com) under the google apps framework options.
7. Follow the instructions on this page: Uploading your application to upload your files to the google app engine service.
8. Here is the code for my masteredsoftware.py file:
   print "Status: 302 Moved"
   print "Location: http://www.masteredsoftware.com/"
   print ""
9. Also setup your app.yaml file pointing to the main script file:
   application: masteredsoftware
   version: 1
   runtime: python
   api_version: 1
   
   handlers:
   - url: /.*
   script: masteredsoftware.py
10. If everything worked out the code files above should forward your naked domain requests to your www.yourdomain.com version.
11. ...Finally you might want to add google analytics to your pages site so that you can track visitors. Just click on the area you want your analytics code, click the edit html link (on the bottom right of the google pages page), surround the analytics code with div tags and save and publish your changes.
12. Now you can play with some of the other cool features that Google apps for your domain allows (such as gmail hosting, etc...)
13. Have Fun domain hosting!
    

VSS to SVN Migration

The last couple of weeks at work I've been moving our Source Control tool from Visual Source Safe (VSS) to Subversion (SVN). There are many reasons to move away from VSS, mainly it just sucks, random corruption, that nice error message about having to run the analyze tool, and just the fact that it is a very poor Microsoft Product. Also VSS doesn't support some of the features we want as we grow such as branching and merging.

There were other Source Control options we investigated when looking to migrate off of VSS, the most logical was to move to Microsoft's Team Foundation Server (TFS), which I've heard is very robust and is an easy switch from VSS, it also apparently has some really nice features for controling Database Schema and Code (Stored Procedure) changes. The problem with TFS was, to work effectively (in addition to Costly TFS licensing) our Visual Studio Developers would have to upgrade to the Visual Studio Team System which is an additional cost. Our decision was to go open source, SVN is a very popular technology for Source Control and it supports many of our desired processes for Source Change Management, such as branching and merging releases.


Here are some helpful links for SVN and the conversion from VSS:
Good Overview of the Upgrade Process

Subversion Main Site

Tortoise SVN Client

VSS to SVN Migration Tool

TamTam SVN Visual Studio SCC Provider Plugin

Since we wanted to retain the check-in history from Visual Source Safe, we used the VSS to SVN Migration tool listed above. I made some modifications to the tool as well, I wanted to keep the user who originally made the change as well as the date the change was made (the 2nd version of the Migration tool just uses one sign-on to SVN and just uses the current computer time for the check-in time).

My change parses the user name out of the Version information from VSS and uses this login instead for SVN, it also parses the date and time the checkin was made, switches the computers clock to that date and time, and then finally does the commit to the SVN repository (which is on my local machine anyways because I couldn't get the the program to work on Windows 2003). I did run into some problems where on some files I couldn't get the versions from VSS for some reason, maybe these files were from a backup/restore in VSS and that is why it had trouble, I'm not sure but I was able to manually import the files that had this problem and the rest of the process completed smoothly.

Here is the direct link to my Modified Version of PowerAdmin's VSS to SVN Migration tool (I sent it to them, but in case they don't repost my version, here it is):
VSSMigrate3_code.zip

The next step was to move my local SVN repository to the final destination server, which can easily be done with the Subversion dump and load commands as described on this page:
How to move a Subversion Repository


Finally, I've setup backup scripts on the Subversion Windows 2003 Server, we had a similar set of backup batch file scripts running to backup our VSS database and FTP the backup file offsite. These batch files were then setup to be run nightly and weekly via the windows scheduler. The new Subversion backup batch files perform a svnadmin hotcopy to make a copy of the repository without taking it offline, then it zip compresses the files from the hotcopy using the 7zip program's command line arguments, it finally creates a remote ftp script that is executed to ftp the compressed backupfiles offsite. Here are the contents of my fancy batch file backup script for Subversion:



FOR /F "tokens=1 delims= " %%i IN ('date /t') DO SET DATE=%%i

cd C:\Program Files\Subversion\bin\

RMDIR /S /Q D:\backup\SVN\current\
MKDIR D:\backup\SVN\current\

DEL D:\backup\SVN\%DATE%_SVN_backup.zip

svnadmin hotcopy D:\svn_repo D:\backup\SVN\current --clean-logs

"C:\Program Files\7-zip\7z" a -tzip -r -y d:\backup\SVN\%DATE%_SVN_backup.zip d:\backup\SVN\current

cd D:\backup\SVN\

> script.ftp ECHO username
>>script.ftp ECHO password
>>script.ftp ECHO cd ~/backup/SVN
>>script.ftp ECHO binary
>>script.ftp ECHO prompt n
>>script.ftp ECHO mput %DATE%_SVN_backup.zip
>>script.ftp ECHO quit
:: Use the temporary script for unattended FTP
FTP -v -s:script.ftp ftp.servername.com




After that developers must rebind their Solutions to SVN, which can be quite a pain, but once it's setup using TamTam, going forward it is smooth opening the solution files.
One tool that helped me out upgrade the solutions (especially VS2003) was Clean Sources Plus which removes source control bindings from the windows explorer right click menu.

ASP.NET Web Application Project Upgrade - Assembly Reference not found

We've recently undergone a conversion to Visual Studio 2005 Service Pack 1 (SP1) for the new Web Application Projects (WAP) they have added.

The Web Application Projects give more control over the files included in a Web Site because now you have a project file similar to the VS 2003 Web Application model.

There are a couple of nice guides online to upgrading a Web Site to a Web Application Project after installing SP1, here is one I followed for the Upgrade:
Scott Gu's Migrating a Web Site to Web Application Project.

Basically, you just create a new web application project, copy in all your pages and other files, then on all the ASPX pages, right click and choose "Convert to Web Application", which creates the filename.designer.cs files necessary for the new Web Application Model.

To simplify the upgrade initially I didn't add namespaces for the separate App_Code files and I just put them in a folder named "Classes".

After that there weren't many more hoops to jump through for the site to successfully compile.

One reason for the upgrade is to make it easier to create a Web Setup (Installer) for the Web Application Projects where making a Setup Project for the original "Web Site" project was a pain.

After creating the Web Setup Project for our Web Application Project, then testing the installer, I kept getting an error:
"The name 'Namespace' does not exist in the current context"
When I viewed the detailed compiler output: "The type or namespace name 'Namespace' could not be found (are you missing a using directive or an assembly reference?)". This was because the dll files for the website were not in the bin folder underneath the web application root folder.

Once I fixed this in the installer (I manually added the bin folder available from the "View -> File System" context menu, and added outputs: Primary Output for each web project and library as necessary).

Overall, I'm much happier with the new (or old) Web Application Projects model in Visual Studio 2005, it gives the developer much more control over the behavior of the website, the included files, simplifies deployment, and is much faster to compile and test.

Read Foreign Region DVD

I recently got back from a 3 week vacation to Spain and Italy, while I was over there we ran out of space on our camera xD drives so we went to an Internet Cafe in Italy and had them burn a DVD of our pictures that were on our 2 GB xD card. After that I nervously formatted that card so that we could continue taking pictures of our trip.

Once I got back to the States and started compiling all our photos (over 2000 of them!) I insert my DVD with my photos into my computer's DVD drive and I hear it clicking and thrashing and my heart drops into my stomach.

When my DVD ROM couldn't read the disk I paniked, I tried to keep my calm; telling myself it was alright, the files were there, I just had to get them off it.

After much searching online for a solution, I realized the problem was my DVD-ROM was the 2nd generation computer DVD Player with this Movie Industry Region protection so the movie industry can control releases of movies within different countries. What a crock!!!

Luckily I was able to find a firmware upgrade tool that allows my model of DVD Rom to be essentially Region-Less (or region 0) so the DVD player doesn't care what region the DVD is actually encoded for.

After I upgraded the Firmware and used this DVD Region Killer s/w to remove the region protection from windows I could finally read my dvd and use windows to get my photos off of it.

Note: do not leave a foreign region DVD in the drive on bootup or you are bound to get hangs as the O/S looks at the drive to boot from.

HttpWebRequest in C# for Web Traffic Simulation, Watch out for Expect 100 Continue and Nagle

Over the past few months we've been busy at work creating our scalable Intelligent Chat application using C# .NET 2.0 and AJAX. To perform load testing against our application we have written a simple Traffic Generating/Load Simulating Windows Forms application which makes Asynchronous HTTP Requests using the HttpWebRequest object, it is pretty straightforward and has been able to surface many problems in our main application. We did come across a big "Gotcha" in our simulator, after looking at the generated network traffic we noticed that in the header of each request the simulator was sending "Expect: 100 - Continue", after which the server would send the 100 - Continue response to the client. Since this was not the correct behavior, we added the following line of code:
System.Net.ServicePointManager.Expect100Continue = false;
As indicated in the following blog posts:
HttpWebRequest and the Expect: 100-continue Header Problem
HttpWebRequest and Expect 100 Continue

After this change however, the headaches started, everything was fine when we tested in our Development environment, IIS on Windows 2003, when we deployed the simulator into the Test environment (as far as I can tell, exactly the same server configuration, etc...), the simulator was so slow that requests would time out almost immediately.

After using Wireshark to view the network traffic in the test environment, we were seeing some strange behavior, it looked like the simulator was sending the header of the POST request, and then waiting for the Acknowledge (ACK) message back (which would take more than 200 milliseconds) from the server before sending the body of the post. Thanks to our brilliant network engineer we were able to determine that the client caused this behavior by using the Nagle Algorithm for it's requests.

That is when we put in one more line:

System.Net.ServicePointManager.UseNagleAlgorithm = false;
In our simulator and it seemed to fix the problem in the test environment, I have no Idea what is different in our development environment, some IIS configuration, maybe the fact that our test environment servers have .NET 3.0 installed? It is still a mystery to me. I figured I would share our experiences with the Expect 100 continue and Nagle Algorithm to perhaps save others some of the same pain we dealt with debugging this issue.

Update September 7th, 2007:
We found the difference in our development environment, the windows 2003 server in our development environment is R2 with Service Pack 1 (sp1) and the windows 2003 server in the test lab is win 2003 R2 with Service Pack 2 (sp2), after viewing further network traces this seems to stem from IIS sending a 100 continue under sp1 (even when we don't send the expect 100 continue message), then what we see is the client waits a long amount of time before sending the ACK message back to the server (I suspect because it has already sent the entire body in the initial POST headers).

This Blog post points to a hotfix that addresses this issue in sp1:
HTTP.SYS, IIS, and the 100 continue

I've put together a simple table to show all the combinations related to 100 - Continue and Nagle, and the results of the test (it was just too hard to keep it all straight in my head):

Results	Server Sends 100 - Continue (Windows 2003 SP1 && XP IIS 5.1)	Client sends Expect: 100 - Continue	Use Nagle Algorythm
Fast (extra traffic sent)	1	1	1
Fast (extra traffic sent)	1	1	0
Fast (didn't send POST body, after the 100 - Contine it has something to send?)	1	0	1
Slow (waits before sending ACK to 100 - Continue)	1	0	0
Fast Responses	0	1	1
Fast (extra traffic sent)	0	1	0
Slow (Nagle causes a ~200ms wait for ACK)	0	0	1
Fast Responses	0	0	0

(Note: A zero in column one means Windows 2003 SP2)