I have been investigating using BOSH with Windows stemcells to help Windows developers have all the delights of BOSH for their deployment lifecycle.
Very quickly in the journey I created bugs that needed debugging, and it wasn’t the same debugging experience for a BOSH/Linux deployment. I got help. Hopefully the debugging tips below are helpful to you when you’ve got an .exe that needs to run on Windows and you want to use BOSH to do it.
Summary documentation for BOSH on Windows is available which was helpful.
The screen images in this blog post are from Windows Server 2012R2 stemcells which I found at https://bosh.io/stemcells
To experiment with BOSH on Windows I created a sample BOSH release https://github.com/cloudfoundry-community/sample-go-windows-boshrelease which borrowed heavily from https://github.com/cloudfoundry-incubator/garden-windows-bosh-release
It compiles a Hello World Golang web server that runs on port 3000. But it timed out when I tried to access the app:
$ curl 10.0.0.5:3000 -v
* Rebuilt URL to: 10.0.0.5:3000/
* Hostname was NOT found in DNS cache
* Trying 10.0.0.5...
* connect to 10.0.0.5 port 3000 failed: Connection timed out
* Failed to connect to 10.0.0.5 port 3000: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.0.0.5 port 3000: Connection timed out
This was deployed on Google Compute Platform (GCP) and I knew my GCP firewalls allowed any app on the bosh-bastion to access any VM in the subnet, including my Windows VM running at 10.0.0.5.
Debugging time.
Fetching BOSH job logs
At the time of this writing, bosh2 logs --follow does not work on Windows VMs to stream the logs of running jobs, however, we still have the static dump version of bosh2 logs:
$ bosh2 logs -d simple-go-web-app
...
22:40:34 | Fetching logs for webapp/b95a2a12-b7c4-4c46-9003-713d64c013c0 (0): Finding and packing log files (00:00:01)
...
Downloading resource 'b174c5b7-da7c-40aa-5926-2af98a6332b2' to '/home/drnic/workspace/sample-go-windows-boshrelease/simple-go-web-app-20170723-224036-02661666.tgz'...
To unpack and view all the logs:
$ tar xfz simple-go-web-app-*; tail -n 200 simple-go-web-app/simple-go-web-app/* simple-go-web-app/*
...
==> simple-go-web-app/simple-go-web-app/job-service-wrapper.out.log <==
[martini] listening on :3000 (development)
...
It seems that the simple-go-web-app.exe compiled Golang app is successfully running and listening on port 3000.
Delete the logs:
rm -rf simple-go-web-app*
A one-line combo I used was:
bosh2 logs -d simple-go-web-app && tar xfz simple-go-web-app-* && tail -n 200 simple-go-web-app/simple-go-web-app/* simple-go-web-app/* | less && rm -rf simple-go-web-app*
These logs showed that the simple-go-web-app.exe was running and it thought it was listening on 3000.
Using RDP to access a Windows VM
Normally to access a BOSH VM we would use bosh2 ssh to get a secure terminal. This does not work with Windows servers. Instead, we must use the Microsoft Remote Desktop Protocol (RDP) [download links].
Currently RDP access and authentication is managed by the cloud provider, not via BOSH.
With luck, you can find the VM on GCP dashboard and click "RDP" button:
I wasn’t lucky.
Using RDP via an SSH tunnel
No matter how many firewalls or gateways or RDP clients I played with I could not get RDP working.
My eternal thanks go to David Jahn who suggested running an SSH tunnel from my local machine to the Windows VM thru the bastion.
To setup an SSH/RDP tunnel with GCP, where 10.0.0.5 is the GCP IP of the VM to be debugged:
gcloud compute ssh bosh-bastion -- -L 3389:10.0.0.5:3389
This will open a normal ssh session into bosh-bastion but will transparently include a tunnel from localhost:3389 to 10.0.0.5:3389.
Using Microsoft Remote Desktop, create a new "desktop":
The username and password will change for each VM. The localhost:3389 will be available whenever you the gcloud compute ssh bosh-bastion -- -L ... command above is running.
To setup a username and password on the Windows VM, click "Set Windows password" and follow the prompts.
Initial look at how BOSH runs on Windows
As a quick sanity test, I confirmed that files and paths were as I expected.
In a PowerShell terminal, run:
cmd /c start /b set
The Path env var were displayed in full:
Path=C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\ProgramData\
GooGet;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\Google\Compute Engine\metadata_scri
pts;C:\Program Files\Google\Compute Engine\sysprep;C:\var\vcap\bosh\bin
The BOSH packages were accessible via the normal folder aliases, within the c:\ drive:
C:\Users\drnicwilliams>dir c:\var\vcap\packages
Directory: C:\var\vcap\packages
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----l 7/23/2017 9:38 PM simple-go-web-app
Firewalls were enabled by default
Using Internet Explorer, after disabling IE default security, I could view the http://localhost:3000 application locally:
This confirmed that the application was running as expected and can be accessed locally.
Therefore the Windows Server must have additional firewall rules that aren’t managed by GCP or BOSH that are blocking ingress traffic to port 3000.
To test, I disabled the private firewalls:
Success. The curl request from bosh-bastion now worked:
$ curl 10.0.0.5:3000
Hello world
Configuring Windows Firewall from BOSH
Manually changing the firewall rules only demonstrated where the issue was. The BOSH release needed a specific firewall rule to allow TCP traffic into port 3000 (or whatever port the application is bound to) every time the release was deployed.
Windows Firewalls can be scripted using PowerShell New-NetFirewallRule command. To run PowerShell using BOSH, one option is to include a bin/pre-start.ps1 script.
New-NetFirewallRule -LocalPort <%= p("port") %> `
-Protocol TCP `
-Direction Inbound `
-Name simple-go-web-app `
-DisplayName simple-go-web-app
See pre-start.ps1 in the final BOSH job.
Other PowerShell hooks are documented https://bosh.io/docs/windows.html
Summary
When deploying Windows Server VMs with BOSH we do lose a couple of standard debugging options we have with traditional BOSH Linux VMs. Instead we can use Microsoft RDP to get a Windows desktop session, as well as download the BOSH job logs using bosh2 logs.
The Windows stemcell has default firewall rules. If your BOSH jobs require additional ingress or egress rules, use a pre-start.ps1 script to run PowerShell commands.
Spread the word
