Tech Notes

Tech Notes Resources StreamCatcher

How to use StreamCatcher to control how IIS logs dynamic WebHub page requests

Required Software

Microsoft IIS - any version
StreamCatcher v1.9+
WebHub Runtime - any version
WebHub Components - any version

StreamCatcher WebSite Config - BEFORE

This would be a common configuration file for a site running WebHub - before using the new logging feature. For information about any of these settings, please see the help file named SCHelpWH.hlp

[cslive.Settings]
URLFiltering=Enabled
SiteRootRequestTreatedAs=WebHub

[cslive.WebHubAppID_SessionAccess]		
csweb=All

[cslive.WebRobotConfiguration]
WebRobotDetection=Enabled 
DailyHitReport=Disabled

[cslive.UserAgentAnalysis]
UserAgentAnalysis=Enabled

[cslive.GlobalReMapping]
RedirectionOfALLRequests=Disabled
MaxAllowedURLLength=1024

[cslive.WebHubURLFiltering]
WebHubURLFiltering=Enabled
DefaultAppID=csweb
OtherFilteredAppIDs=hubvers
AppIDsThatSendBlankSessionsToWebRobots=csweb
DefaultRunner=/scripts/runisa.dll
UseURLRunnerIfPresent=true
DefaultReMapping=Enabled
DefaultReMappingSpec=/*=%r?%a:%p[:%s][:%c]
DefaultReMappingSpecForWebRobots=/*=[%r?][%a:]%p[:%o][:%c]
Separator=:
ReplaceURLColonsWithSlashes=No
WebRobotSession=1001
SendPermanentlyMovedStatusCodeIfNotWebRobotSession=Yes

StreamCatcher WebSite Config - AFTER

If you copy this for your own use, be sure to change cslive to your own WebSiteID and csweb to your own AppID.

Everything about LandingPage is necessary if and only if you want to have special LandingPages which are always used to receive visitors coming in from an advertiser such as google.

[cslive.Settings]
URLFiltering=Enabled
SiteRootRequestTreatedAs=WebHub
RemappingSyntax=RegEx

[cslive.WebHubAppID_SessionAccess]	//unchanged]

[cslive.UserAgentAnalysis] //unchanged]

[cslive.GlobalReMapping //unchanged]

[cslive.WebHubURLFiltering]
WebHubURLFiltering=Enabled
DefaultAppID=csweb
OtherFilteredAppIDs=csorder,hubvers
AppIDsThatSendBlankSessionsToWebRobots=csweb,csorder
DefaultRunner=/scripts/runisa.dll
UseURLRunnerIfPresent=true
DefaultReMapping=Enabled
DefaultReMappingSpec=/*=%r?%a:%p[:%s][:%c]
DefaultReMappingSpecForWebRobots=/*=[%r?][%a:]%p[:%o][:%c]
Separator=:
ReplaceURLColonsWithSlashes=No
RegExClassicRemapToSpec=%r?%a:%p[:%s][:%c]
RegExClassicRemapToSpecForWebRobots=[%r?]{%L/}[%a:]%p[:%o][:%c]
WebRobotSession=1001
SendPermanentlyMovedStatusCodeIfNotWebRobotSession=Yes
DefaultLingvo=eng
LogPageExtension=.whpage

[cslive.RegExMacros]
«session»=([0-9]+(\.[0-9]+)?)?
«appid»=([^/:]*)
«page»=([^/:]*)
«command»=(.*)

[cslive.WebHubURLsToRemapWithinDomain]
SCSeparator===>
^/«session»\?(.+)==>r=,L=,a=csweb,p=,s=$2,c=$3 (~Logging content="custom" customSection="WebHubStandardLogging2"~)
^/((:«session»)?)?$==>r=,L=,a=,p=,s=$3,c= (~Logging content="custom" customSection="WebHubStandardLogging1"~)
^/«appid»([:/](«page»([:/]«session»([:/]«command»)?)?)?)?$==>r=,L=,a=$1,p=$4,s=$6,c=$9 (~Logging content="custom" customSection="WebHubStandardLogging2"~)
^/«page»([:/]«session»([:/]«command»)?)?$==>r=,L=,a=,p=$1,s=$3,c=$6 (~Logging content="custom" customSection="WebHubStandardLogging2"~)

[cslive.WebHubStandardLogging1]
Match1=Target:${whSession} Pattern:^.+$ URIStem:/${whAppID}/ URIQuery:session=${whSession}
Default=URIStem:/${whAppID}/ URIQuery:

[cslive.WebHubStandardLogging2]
Match1=Target:${whSession}---x---${whCommand} Pattern:^.+---x---.+$ URIStem:/${whAppID}/${whPage} URIQuery:session=${whSession}&cmd=${whCommand}
Match2=Target:${whSession} Pattern:^.+$ URIStem:/${whAppID}/${whPage} URIQuery:session=${whSession}
Match3=Target:${whCommand} Pattern:^.+$ URIStem:/${whAppID}/${whPage} URIQuery:cmd=${whCommand}
Default=URIStem:/${whAppID}/${whPage} URIQuery:

[cslive.DynamicURLsToReMapWithinDomain]
SCSeparator===>
^/landingpage([^/:?]*)\?(.*)$==>/scripts/runisa.dll?csweb:landingpage$1::$2 (~Logging content="custom" customSection="LandingPageLogging"~)

[cslive.LandingPageLogging]
Default=URIStem:/csweb/landingpage$1.whpage URIQuery:cmd=$2

Resulting IIS Log Flie - Sample

In the following log, notice that dynamic pages (?csweb:contact) are logged as static pages (/csweb/contact.whpage), and the session number, when known, is logged as part of the URI Query.

The reason for doing this is to make the resulting log files more human-readable and, probably, more likely to suit off-the-shelf log analysis software.

#Software: Microsoft Internet Information Services 5.0
#Version: 1.0
#Date: 2009-08-30 06:01:03
#Fields: date time c-ip cs-username cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes cs-host cs(User-Agent) cs(Cookie) cs(Referer) 
2009-08-30 06:01:03 123.123.12.12 - GET /csweb/landingpagehome.whpage cmd=gclid=123xyz 200 12374 629 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:08 123.123.12.12 - GET /csweb/landingpagehome.whpage - 200 12374 590 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:16 123.123.12.12 - GET /csweb/landingpagerates.whpage - 200 395 591 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:21 123.123.12.12 - GET /csweb/ - 200 12338 575 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:22 123.123.12.12 - GET /csweb/contact.whpage session=483320437 200 10987 624 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:22 123.123.12.12 - GET /csweb/shipmentrate.whpage session=483320437 200 9274 629 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:24 123.123.12.12 - GET /csweb/servicesdesc.whpage session=483320437 200 5209 629 www.csweb.biz Mozilla/5.0+
2009-08-30 06:01:24 123.123.12.12 - GET /csweb/login.whpage session=483320437 200 3708 622 www.csweb.biz Mozilla/5.0+

Remember, configuration can be tested on a non-production machine using the StreamCatcher Console (SCConsole.exe) and a spreadsheet, and that is definitely recommended!

See also: FBPrepLog routine for importing log files into Firebird SQL.

contributed by Ann Lynnworth on 30-Aug-2009

Ask a question or make a suggestion about this topic