Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We moved an application to a different server and turned off the site in IIS on the old server. It looks like there are still web crawlers (Googlebot, Baidu, etc.) that are trying on the old IP address, so they get a 404 error. We would prefer them to get a 503 error so that they will try again later, which will give the DNS more time to propagate. Is there an easy/straightforward way to do this with IIS6?

share|improve this question

4 Answers 4

You could create a "catch-all" page and add the following asp code:

context.context.Response.StatusCode =
(int)HttpStatusCode.ServiceUnavailable;
share|improve this answer
    
Turns out they were accessing it by IP. I set up a slightly more complicated script that would issue appropriate 301 redirects to the correct site by name instead of IP address. –  yakatz Jan 17 '12 at 22:36

I'm migrating several websites to a new server tonight tonight, from IIS 6 (Windows 2003 server) to IIS 7.5 (Windows 2008 Server), with changing IP addresses, and was worried about web crawlers as well. A friend pointed out that if I take the application pool offline on the old server, and leave the websites up, the sites all return 503 error codes, even if you try to access them using the IP addresses, until the DNS has propagated. Seems to be working great.

(I know this is a late post, but I spent hours searching for a solution just like this, so I thought I'd add it here.)

share|improve this answer
up vote 0 down vote accepted

This works in IIS 6 and IIS 7.
This is for my particular store (ASPDotNetStorefront), modify as you see fit.

web.config

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.web>
        <customErrors defaultRedirect="/404handler.asp" mode="On" />
    </system.web>
</configuration>

404handler.asp

<%@ Language=VBScript %>
<%
Option Explicit

Sub Redirector()
    Dim strRequest
    Dim strRedirect
    Dim strServer
    Dim objRegExp
    Dim tokens

    strServer = "http://mynewurl.example.com"

    strRequest = Request.QueryString("aspxerrorpath")
    If StrComp(strRequest, "") = 0 Then
        strRequest = Request.QueryString
        strRequest = Mid( strRequest, Instr( Instr(1, strRequest, "://") + 3, strRequest, "/"))
    End If

    Set objRegExp = New RegExp
    objRegExp.IgnoreCase = True

    'Product Page
    objRegExp.Pattern = "^/p-[0-9]+-.*\.aspx$"
    If objRegExp.Test(strRequest) Then
        strRedirect = strServer + strRequest
        Call DoResponse(strRedirect)
    End If

    'Product Print Page
    objRegExp.Pattern = "^/print-([0-9]+)-(.*)\.aspx$"
    Set tokens = objRegExp.Execute(strRequest)
    If tokens.Count > 0 Then
        strRedirect = strServer + "/p-" + tokens(0) + "-" + tokens(1) + ".aspx?print=1"
        Call DoResponse(strRedirect)
    End If

    'Product Email Page
    objRegExp.Pattern = "^/email-([0-9]+)-(.*)\.aspx$"
    Set tokens = objRegExp.Execute(strRequest)
    If tokens.Count > 0 Then
        strRedirect = strServer + "/p-" + tokens(0) + "-" + tokens(1) + ".aspx"
        Call DoResponse(strRedirect)
    End If

    ' Signin Page
    objRegExp.Pattern = "^/signin.aspx(\?.*)?$"
    Set tokens = objRegExp.Execute(strRequest)
    If tokens.Count > 0 Then
        strRedirect = strServer + strRequest
        Call DoResponse(strRedirect)
    End If

    'Manufacturer Page
    objRegExp.Pattern = "^/m-[0-9]+-.*\.aspx(\?.*)?$"
    If objRegExp.Test(strRequest) Then
        strRedirect = strServer + Replace(strRequest, "CatID", "FilterID", 1, -1, vbTextCompare)
        Call DoResponse(strRedirect)
    End If

    'Category Page
    objRegExp.Pattern = "^/c-[0-9]+-.*\.aspx(\?.*)?$"
    If objRegExp.Test(strRequest) Then
        strRedirect = strServer + Replace(strRequest, "ManID", "FilterID", 1, -1, vbTextCompare)
        Call DoResponse(strRedirect)
    End If

    'Other ASPX Pages
    objRegExp.Pattern = "^/.*\.aspx(\?.*)?$"
    If objRegExp.Test(strRequest) Then
        Dim Message, a
        Message = "Original request was: " + strRequest + vbCrLf + vbCrLf + "Info:"
        Message = Message + vbCrLf + "  " + Request.ServerVariables("ALL_HTTP")
        Message = Message + vbCrLf + "  Client: " + Request.ServerVariables("REMOTE_ADDR")
        Call SendEmail("me@example.com", "no-reply@oldserver.example.com", "Missed ASPX 404 Error", Message)
        'Response.Write(Message)
    End If

    ' Ignore errors for non-aspx pages
End Sub

Sub SendEmail(seTo,seFrom,seSubject,seBody)
    on error resume next
    Dim EmailObj, iConf, Flds
    Set EmailObj = Server.CreateObject("CDO.Message")
    Set iConf = CreateObject("CDO.Configuration")
    Set Flds = iConf.Fields
    Flds.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
Flds.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "localhost"
    Flds.Update
    EmailObj.Configuration = iConf
    EmailObj.From = Chr(34) & seFrom & Chr(34) & Chr(60) & seFrom & Chr(62)
    EmailObj.ReplyTo = Chr(34) & seFrom & Chr(34) & Chr(60) & seFrom & Chr(62)
    EmailObj.Subject = seSubject
    EmailObj.Fields.Update
    EmailObj.TextBody = seBody
    EmailObj.To = Chr(34) & seTo & Chr(34) & " <" & seTo & ">"
    EmailObj.Send
    Set EmailObj = nothing
    on error goto 0
End Sub

 Sub DoResponse(strRedirect)
    If StrComp(strRedirect, "") <> 0 Then
        Response.Status = "301 Moved Permanently"
        Call Response.AddHeader("Location", strRedirect)
        Call Response.Write("<html><head><title>Page moved</title></head>")
        Call Response.Write("<body><h1>Page Moved</h1>")
        Call Response.Write("This page may be found at <a href=""" & strRedirect & """>" & strRedirect & "</a>.")
        Call Response.Write("</body></html>")
        Call Response.End
    End If
End Sub

Call Redirector()
    Response.Status = "404 File not found"
%><?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
    <head>
    <title>The Page Cannot be Found</title>
    <meta name="robots" content="noindex"></meta>
    <style>
    <!--
    BODY {
        FONT-FAMILY: Verdana, Arial, Helvetica, sans-serif
    }
    H1 {
        COLOR: #cd5c5c
    }
    H2 {
        COLOR: #cd5c5c
    }
    TABLE {
        WIDTH: 540px
    }
    -->
    </style>
    </head>
    <body>
        <table>
            <tr>
                <td>
                <h1>The Page Cannot be Found</h1>
                <hr />
                <h2>Potential Causes:</h2>
                <ul>
                    <li>The requested file has been deleted.</li>
                    <li>The requested file has never existed.</li>
                    <li>The requested file is temporarily unavailable  due to maintenance, upgrades, or other similar reasons.</li>
                    <li>The requested file has moved, and we didn't automatically forward to the new page.</li>
                </ul>
                <hr />
                <h2>Please try the following:</h2>
                <ul>
                    <li>If you typed the page address into the web browser, make sure that you spelled it correctly.</li>
                    <li>Open the <a href="http://mynewurl.example.com/">home page</a> of this site.</li>
                    <li>Click the Back button to try another link.</li>
                </ul>
                <hr />
                <h2>HTTP 404 - File not found</h2>
                </td>
            </tr>
        </table>
    </body>
</html>
share|improve this answer

You know what would be much easier? Put this in /robots.txt for as long as you don't want your sites crawled by search engine bots:

User-Agent: *
Disallow: /
share|improve this answer
    
We did not want to not have the site indexed at all, we just wanted the content at the new IP only to be indexed. –  yakatz Jul 31 '12 at 17:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.