Test Post

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Home Lab – Part 2: A Tale of Storage Woe

In Part 1 of this series I wrote about the hardware I used to build two virtual machine hosts on a budget.  The two hosts boot from USB thumb drives  and do not contain any other storage.  My plan was to use my existing file server for virtual machine storage.

FreeNAS File Server

Over a year ago I built a file server to store media files.  I had been hearing and reading a little bit about the ZFS file system and I was looking to build a server that would have ZFS available.  I settled on FreeNAS as the software solution to run on the server.  Unfortunately, I did not realize at the time that ZFS had such high memory requirements.  My hardware follows:

The motherboard I selected only accepted 2GB of RAM, but as I began reading the FreeNAS documentation I quickly learned that 8GB is considered a minimum and that a rule of thumb is that 1GB of RAM is needed for every terabyte of ZFS storage.  It had not occurred to me that a file system could be so memory intensive!  However, given my use case I did find some examples of people who were running ZFS with low memory and I pressed forward.
Continue reading Home Lab – Part 2: A Tale of Storage Woe

Strange Network Delay


A customer site I worked at was using Windows XP clients to connect to a server running Windows Server 2003.  This server hosted a SQL Server database and also had a regular Windows share that held the executable and DLL’s for an application.  The clients all had shortcuts on their desktops to launch the application that looked something like this:


Users would use the shortcut to access the application which would launch and connect to the database.  This all worked fine up until the customer’s hardware got to a certain age and was replaced with new hardware running Windows Server 2008 R2.  As a best practice the Windows Firewall was enabled and only the necessary ports were opened (SMB – TCP port 445; SQL – TCP port 1433).

Network Delay

After the database and application were migrated to the new hardware the users noticed that when accessing a search dialog in the application, it would intermittently hang for a while (hourglass) and then resume.

Troubleshooting an intermittent issue is the very definition of insanity so I first set to work to find out how to recreate the issue consistently.  After interviewing a number of users it seemed that the delay only occurred after the application had been idle and the user had returned to it to perform a search.  Some testing on my part showed that if the application was left idle for 15 minutes, a 40 second delay occurred when attempting to access the search dialog.  Opening the search dialog was instant after this, until the application had been idle for another 15 minutes.


Now that the issue could be recreated consistently I turned to determining the cause.  A quick look at Windows Task Manager showed that while recreating the issue there wasn’t anything amiss with CPU utilization or memory usage so I quickly ruled out resource exhaustion on the client.  My next stop was my trusty old friend Process Monitor.  Here is a screenshot of the trace while recreating the issue (click for a larger image):


The first highlighted area shows a NetWareRedirector error (BAD NETWORK NAME) followed by what looks to be a name resolution process looking at registry entries for Novell NetWare, TCP/IP, NWHOST, and finally the hosts file.  The second highlighted area shows a time gap of approximately 40 seconds and finally in the third area we see the client successfully access a DLL on the Windows network share.

Up until this point I had been suspecting a performance issue with the database since this application was heavily database driven.  However, the Process Monitor trace showed that the delay occurred when attempting to access a DLL on the network share (this particular DLL is the English language resource file for the application so I assume the application is loading the text labels for the dialog).  The next step was to capture a network trace using Wireshark (click for a larger image):



At this point it seemed fairly clear what was happening: the application was trying to access one of its DLL’s on the network share using SMB, there were some SMB errors showing, there was a 40 second delay, and then successful connection to the share.  The NetWare Redirector error was also on my mind as I knew the client machines had the Novell Client installed and had drives mapped to Novell shares in addition to the Windows network share they accessed on the Windows server.  Now I started to research what could cause a delay when accessing a network share that only occurred after an idle period (and in a mixed Windows/Novell environment).  Unfortunately, there are many, many configurations that can cause this issue to manifest!

Let Me Count the Ways …

Some Progress

Because anti-virus software has proved to be the cause of many performance issues I have investigated and the previous investigation turned up SEP as a possible issue I uninstalled it and rebooted the server, but it did not resolve the issue.  Symantec Firewall/Network Threat Protection was not installed at this customer site as I had resolved a very similar issue (40 second delay in this same application) at a previous customer site by disabling that Symantec component.  Since this solution that I had previously used could be not be implemented here I started going through these KB articles and blogs, applying changes, testing and then reverting.  None of the suggested workarounds resolved the issue.  I was not able to try very many changes on the Novell client since my company did not control the Novell infrastructure.

A colleague of mine was troubleshooting the same issue and ended up disabling the Windows Firewall completely which did resolve the issue!  This was not a permanent resolution, however, as we try hard not to purposefully disable security features on production servers.  Disabling the firewall fit some theories as my research and experience showed issues with the Symantec Firewall/Network Threat Protection and the Windows Firewall (WebDAV) but raised other questions.  Usually I would expect a firewall issue to block or allow traffic, not to introduce a delay. It did explain why the issue only appeared after the hardware and operating system were upgraded: Windows Server 2003 firewall is off by default, Windows Server 2008 R2 firewall is on, so likely no software firewall had been present before.

A Resolution

I confirmed that disabling the Windows Firewall resolved the issue and enabling it re-introduced the delay.  I thought it would be straightforward to enable logging on the Windows Firewall, inspect the packets that were being dropped and determine a new rule to add to the firewall to resolve the delay when accessing the network share.


However, after reproducing the issue the firewall log showed no packets or connections denied.  Every single line in the log file had a action of ALLOW.  I don’t know if the Windows Firewall logging doesn’t work correctly or if I didn’t configure it correctly – it looked fairly straight forward.

At this point we had spent so much time on the issue and exhausted our investigative options, we opened a ticket with Microsoft. A support technician looked at the Wireshark traces I provided and was able to find connections to the server on TCP port 524 using the protocol NCP (NetWare Core Protocol).  These connections were not being refused (with a reset or RST packet) but instead received no response at all, causing retransmissions (click for a larger image):


I enabled the Windows Firewall and added a rule to allow TCP port 524, then using the ability to consistently recreate the issue I tried to reproduce it but could not. At this point we could consider the issue resolved!

[Update 10-16-2013] It turns out that adding a firewall rule to allow TCP port 524 did not resolve the issue.  When confirming the fix, something must have touched the share during the 15 minutes of idle time; causing the issue to not reproduce.  I confirmed via network trace that even after the new firewall rule was added that the client still saw retransmissions and a 40 second delay when attempting to contact port 524 on the server.

I did some additional research about why Windows would not respond with a reset (RST) packet when the client tried to connect since we were not actively blocking port 524.  That led me to this MSDN article about Stealth Mode:


Stealth mode is a mechanism in Windows Firewall that helps prevent malicious users from discovering information about network computers and the services that they run. …  Stealth mode blocks outgoing ICMP unreachable and TCP reset messages for a port when no application is listening on that port.

Interestingly, the article notes that network packets dropped by Stealth Mode are not logged!  Stealth Mode can be disabled by this process:

  1. Create the appropriate registry key based on the firewall profile your NIC is associated with.  You may need to create the WindowsFirewall key and the sub keys corresponding to the firewall profile. For example:
  2. Restart the Windows Firewall service

At this point I found that Windows would send the reset packet when trying to connect to TCP port 524.  The Novell Client still retransmits the connection attempt 3 times, but they all receive a reset packet so the process is over in one second (click for a large image):


This was true even when removing the rule to allow traffic to TCP port 524.  So the final solution was not to add a firewall rule, it was to disable Stealth Mode.

[/Update 10-16-2013]

Lessons Learned

  • Ensure that the issue can be consistently recreated
  • Firewalls can introduce delays by blocking traffic that is not essential to the operation, e.g. lookups
  • Pay attention to IP Conversations in Wireshark/Network Monitor and look for retransmissions

Home Lab – Part 1: Hardware

Every IT Pro who wants to accelerate their learning and keep up with new developments in technology should have a home lab.  Being able to standup and configure new software provides a ton of benefits:

  • practical hands on experience
  • learning in your lab can spark ideas used to solve problems in your day to day
  • hone your skillset beyond technologies you use at work
  • allow studying for certifications
  • increase your value to your employer and open up new opportunities

I wanted to setup a home lab for these reasons but stay within as small a budget as possible.  After doing quite a bit of research I was getting discouraged as I saw most home labs were coming in at $1000+.  Many recommendations for lower cost setups mentioned buying previous generation servers off of eBay but those servers were almost universally bulky, loud, hot, and drew lots of power.  Finally, I came across a discussion on the Ars Technica forums where I found a recommendation about using a low-cost embedded motherboard.

Using two of these motherboards for virtual machine hosts I was able to build a lab with two hosts, each capable of running 6 to 8 virtual machines.  The hosts are silent and draw very little power.  They won’t break any speed records but are perfectly suited to learning.  So, thanks to the modern miracles of virtualization and low power hardware a modest home lab can be built on a budget!


Motherboard (with embedded Intel Celeron 1.1 GHz processor)

Gigabyte GA-C847N



Crucial 2x8GB memory sticks


Power Supply

Thermaltake 430 Watt


I put these components into two Antec cases I had laying around from previous builds and used a USB thumb drive on each host to boot the hypervisor.  I did not purchase any storage to put in these hosts as I already had a file server capable of presenting storage as iSCSI targets.  More on that in another post.  Otherwise, one or two SATA drives for each host would suffice for low intensity workloads.



VMware vSphere ESXi 5.1


VMware has the largest market share and is what I see most when working with customers.  Not only will running ESXi for my home lab enable working with multi-tier applications but it allows getting familiar with the virtualization platform.  At some point I will also add Hyper-V to the mix.  At first probably as a nested hypervisor on one of the ESXi hosts, but in the long run on its own host.


After building this home lab I can comfortably run 6 to 8 VM’s on each host.  I’m running two domain controllers on Windows Server Core and the vCenter Server Appliance as well as a VM with SQL Server.  I had a PC running Crashplan that I virtualized and it is currently putting the most load on the host.  Here is the CPU graph from the host with the Crashplan server.



I was having trouble setting up an ODBC connection to an Oracle database. Then I learned about tnsping from a colleague. Providing tnsping with an IP address attempts to connect on port 1521 (the default) an returns OK if successful. Even better, if you provide tnsping with a name it will attempt to locate the name in the TNSNAME.ORA file and resolve it to connect.

So, using tnsping with the name I had in the ODBC dialog showed that it couldn’t be located in the TNSNAME.ORA file and it was straight-forward from there to resolve the issue. The main issue was that a period separated name in the TNSNAME.ORA file (e.g. ABC.XYZ) showed up the TNS Name drop-down with only the first part (e.g. ABC).

Once I determined that the name was incorrect and verified the full name, it was easy to type in the missing part and successfully connect.

Notepad++: Delete Trailing Empty Line

I had the need today to delete the last empty line from a bunch of text files. There are a lot of pointers on Google to using the Extended search/replace in Notepad++ with \r\n (for files with Windows line-endings). However, that search/replace removes all of the CRLF line endings and you end up with a file that is all on one line.

It seems that as of Notepad++ 6.0, \r\n works when using Regular Expression search/replace. So I ended up with

Find what: \r\n^$
Replace with:
Search Mode: Regular Expression

That removed the last blank line quite nicely.