smithvoice.com
 Y'herd thisun? 

“'Everyone wants to eat like an American on this globe,' said Daniel Basse of AgResource, a Chicago consultancy. 'But if they do, we're going to need another two or three globes to grow it all.'”

from As world's appetite soars, so do food bills by David Streitfeld

MONO/Ubuntu part 16 - Logging Apache Accesses and Errors

TaggedASP.Net, MONO, Linux

You don't want to expose a web server to the wild without setting up logging. We all know this. We sometimes forget this. Don't.

Windows developers over the years get cushy with IIS's protection and we take it for granted, but now we're on Apache and going back to our roots of not trusting the core is a good thing.

Our current virtual hosts file is quite a bit different from the Apache default file, if we look to that original we'll see the recommended minimum log lines.

nano /etc/apache2/sites-available/default

Open your site's vhost file with sudo nano, scroll down to just above the closing </VirtualHost> tag and add those exact same lines. Save the file and reload Apache.

 

Full-on and Thinned-useful Access Logs

Make a few hits to correct urls to your server, then open your access.log with nano /var/log/apache2/access.log.

Below you see the hits to our demo's default defpage.aspx (in the log as a "/" slash because the request was for the root of the website), followed by a direct hit on the inline.aspx.

 

172.121.0.15 - - [23/Apr/2009:10:58:10 -0700] "GET / HTTP/1.1" 200 780 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Tablet PC 2.0; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618; WWTClient2)"
172.121.0.15 - - [23/Apr/2009:10:58:18 -0700] "GET /inline.aspx HTTP/1.1" 200 738 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Tablet PC 2.0; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618; WWTClient2)"

 

Out of the box format of the logging is pretty good. Showing the IP Address of the remote client, two dashes for values from a more trusting age that are now not provided by clients for security reasons, the date of the hit, the type of hit ("GET" or "POST", usually "GET"), the requested file, the Protocol, the result status (200 being completed, your custom redirects will also show up as 301s), the number of bytes transfered to the caller, the Referrer (if the caller did not come from clicking a link on another page you'll see the quoted dash "-") and at the end the Browser/Machine details.

For the demo web app of simple textual pages, the log is nice and direct. Things change though when you start adding graphics, styles and javascript. Take a gander at the default logging for a single user request for one of our smithvoice.com pages.

According to the referrer value, this user (I xxx-ed out their actual IP) used their 32bitIE-8 searchbox on a 64bit Windows Vista machine to hit google with the query "integers only in text box for telephone" and clicked the resulting google link to our page

 

 

66.72.xxx.xxx - - [26/Mav/2009:12:25:42 -0700] "GET /scr/sssvrs.js HTTP/1.1" 200 6712 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /sty/nvsty.css HTTP/1.1" 200 1595 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /sty/mstr.css HTTP/1.1" 200 7500 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:41 -0700] "GET /cs-integer-textbox.aspx HTTP/1.1" 200 32609 "http://www.google.com/search?q=integers+only+in+text+box+for+telephone&rls=com.microsoft:en-us:IE-SearchBox&ie=UTF-8&oe=UTF-8&sourceid=ie7&rlz=1I7GGLL_en" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /scr/nvtr.js HTTP/1.1" 301 269 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /scr/rcsnvt.js HTTP/1.1" 200 6319 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/sv2004.gif HTTP/1.1" 200 2091 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/srchbutnormal.gif HTTP/1.1" 200 1137 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/boxed-bg.gif HTTP/1.1" 200 43 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/dummy.gif HTTP/1.1" 200 43 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/box-tl.gif HTTP/1.1" 200 189 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:42 -0700] "GET /ims/panel-bg-tl3.gif HTTP/1.1" 200 96 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-t.gif HTTP/1.1" 200 65 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/panel-bg-tr3.gif HTTP/1.1" 200 95 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-tr.gif HTTP/1.1" 200 188 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/panel-bg-bl3.gif HTTP/1.1" 200 96 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-l.gif HTTP/1.1" 200 65 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/panel-bg-br3.gif HTTP/1.1" 200 96 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_top.gif HTTP/1.1" 200 54 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/panel-grad3.gif HTTP/1.1" 200 99 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /trms/pointer.png HTTP/1.1" 200 368 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/srchbox.gif HTTP/1.1" 200 1236 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/bread-bg-l.gif HTTP/1.1" 200 139 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/bread-bg.gif HTTP/1.1" 200 55 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/bread-bg-r.gif HTTP/1.1" 200 888 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_topleft.gif HTTP/1.1" 200 317 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_topright.gif HTTP/1.1" 200 226 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /trms/box.gif HTTP/1.1" 200 847 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_middleleft.gif HTTP/1.1" 200 51 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_middleright.gif HTTP/1.1" 200 46 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_bottomleft.gif HTTP/1.1" 200 166 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_bottomb.gif HTTP/1.1" 200 48 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/tb_bottomright.gif HTTP/1.1" 200 163 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/footer-bg-grad.gif HTTP/1.1" 200 260 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/footer-bg-tl.gif HTTP/1.1" 200 99 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/footer-bg-tr.gif HTTP/1.1" 200 99 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/footer-bg-bl.gif HTTP/1.1" 200 99 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/footer-bg-br.gif HTTP/1.1" 200 98 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-r.gif HTTP/1.1" 200 65 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-bl.gif HTTP/1.1" 200 190 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-b.gif HTTP/1.1" 200 65 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /ims/box-br.gif HTTP/1.1" 200 189 "http://www.smithvoice.com/cs-integer-textbox.aspx" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"
66.72.xxx.xxx - - [26/Mar/2009:12:25:43 -0700] "GET /favicon.ico HTTP/1.1" 200 1406 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.30618; .NET CLR 3.5.30729)"

 

OUCH! That's one user hit on one relatively small page. Because the page has rounded corner images, our site logo, our favicon, css, external javascript and so on and because the default is to log every single one of them being served the log becomes a bog.

Each of thes resources does have to be accessed by the remote user but not every one of them has to be logged. Doing so kills disk space fast and makes getting information out of the logs nearly impossible. So let's change the logging options using the optional Apache module mod_setenvif.

First, turn on that module:

sudo a2enmod mod_setenvif

Second, open up the vhost file...

sudo nano /etc/apache2/sites-available/asdf

Scroll back down to the log section and change the default CustomLog line to these bolded lines

ErrorLog /var/log/apache2/error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
  
CustomLog /var/log/apache2/access.log combined  env=!dontlog
SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(js)|(ico)|(css)$" dontlog

The power comes with the bit tacked on the CustomLog line, "env=!dontlog" tells the module to expect a list of things not to log. Our list is the next line. Requests for files ending in the extensions in the parenthesis are in the "SetEnv" ... "dontlog" set (we showed the optional case-insensitive SetEnvNoCase command, if you don't care about case sensitivity just use "SetEnv"). If you want to use other extensions just follow the same pipe/parenthesis pattern.

Save and close the file, restart Apache and check your access.log again. You should see the request for the page and all of its important information but all of the automatic graphics, js and css requests no longer are creating noise and weight to the file.

A few more things...

1) On one server, I noticed an immediate drop in the process count and memory useage after turning on the above filter. Going from 400+ processes on the server after a few days of Apache uptime to a stable hundred or so with weeks of uptime. Maybe it was something else, but all I can remember changing was that.

2) There are lots of other tricks you can do with your access logging, you can change the format, filter out search engines and bots, and more. Yahoogle to make them efficient and effective for you.

3) Now and then you may want to comment out the filtering line and let full logging happen, then spend a coffee break scanning the files to see if there's anything odd happening -- just a paranoid thought.

Love your Error.log

Doing a dontlog on the error.log is typically not a good thing, if anything you want more error information instead of less. But watching your error.log can lead to ways to make it less verbose.

For instance, exposing our current demo site to users will instantly start filling the error.log with 404 FileNotFound lines for a /favicon.ico, and if the site is opened to the internet then there will be loads of similar lines for a missing robots.txt file.

Favicon.ico is the little site image you see in your browser tab/addressbar. Originally only called up by IE5 when a human added your page to their favorites list, now most requesters automatically look for this file in your site root. Rather than fighting it, just create such a file using any of the many online tools and pop it in your root folder. That 404 noise will go away (and your site will look nicer in browsers).

robots.txt is called up by rule-following search engines and spider bots, its a simple file they use to find out what you don't want indexed. You can deny indexing of certain folders, deny specific bots or all bots completely, there are a bunch of options. All you have to do is open notepad/nano and create the file with "User-Agent: [value]" followed by the rules you want them to play by.

Here's an example of a robots.txt that specifically tells the Google Images search engine to not grab any graphics from your site and then tells All bots to not index the files in your site subfolders "mystuff" and "mywifesstuff":

User-agent: Googlebot-Image
Disallow: /
User-agent: *
Disallow: /mystuff/
Disallow: /mywifesstuff/

Pretty simple, for more variations and details hit up robotstxt.org.

Save the file to your site root and now those 404s stop showing up in your error.log and what's left are things that you can really start using to tighten up your site.

Quickie to save you time while devtesting: Tail

While you're working on stuff it gets to be a drag to have to keep opening the logs files in nano and CTL-Ving down to the bottom to see the changes. On Windows there are a lot of options, you might have used WinTail or Tail4Win to show changes to text files as they happen ... on 'nix, it's part of the os and logically it's just called "tail".

Because the Ubuntu server interface is one big "window", tailing a file takes up the whole screen so doing it when physically logged onto a box isn't a big help. But if you're on the box remotely it's great. Just open a new windowed SSL or PuTTY connection and call it up with "tail -f [file name]" as in:

tail -f /var/log/apache2/access.log

Hit a page and watch the events. CTL-C will stop the tailing.

 


Pretty much, that's it. We've covered the basics and hopefully most of the gotchas for a Windows developer to go from being a Nonix to running ASP.Net safely on Ubuntu server. Now you're off to typing good old .Net code and having it all when your company says they can't currently afford additional Windows Server licenses.

All that's left is a few helpful workaday tools, then a rundown of how to install and get working on a world-class database server that even a tough corporate manager will allow.


jump to:

  • 1: Why?
  • 2: Installation
  • 3: Update the OS with APT
  • 4: Remoting to the box with SSH
  • 5: NANO quickies for 95% of the jobs
  • 6: Firewalling Ubuntu 8.10 Server
  • 7: Installing Apache2 and MONO
  • 8: Test client host files
  • 9: Configure MONO on Apache2
  • 10: Apache default pages
  • 11: Handling Apache and Mono Errors
  • 12: Subdomains and Christian porn
  • 13: Virtual Host Tweak: unmanaged cAse sensitiviTy
  • 14: Managed cAse sensitiviTy
  • 15: Managing files and folders
  • 16: Logging Apache Accesses and Errors
  • 17: Ubuntu Task Managers
  • 18: Ubuntu Services and Service Managers
  • 19: Installing Oracle 10g XE Server
  • 20: Connecting Mono to Oracle


  • home     who is smith    contact smith     rss feed π
    Since 1997 a place for my stuff, and it if helps you too then all the better smithvoice.com