OIT VM Hosting - VM Maintenance

Billing Information

Billing information can be located in Exchequer.

Installing Applications

The ability to install packages for Linux VMs is not enabled by default.  To install applications on a hosted Linux VM or to have this ability enabled, enter a ServiceNow Request with Systems-UNIX-OIT.  Application installation on Windows hosts is the responsibility of the owner.

Rebooting

To request a reboot for a hosted VM enter a ServiceNow Request with either Systems-Windows-OIT or Systems-UNIX-OIT as appropriate. 

Patch Monkey

Patch Monkey is the internally developed patch automation tool that is used to patch both Windows and Linux hosts (Note - this does not apply to hosts that are self-managed). Patch Monkey schedules and executes scripts on the host OS's that do the actual patching and verification. Patch Monkey creates a scheduled maintenance window in the monitoring application (Spectrum) just before it reboots a VM so no alarms are generated. The average maintenance window lasts 15 minutes for Linux hosts and 30 minutes for Windows hosts, but could extend to 4 hours depending on what is being patched. The maintenance windows are entirely time based; Patch Monkey creates them with a time limit and Spectrum does the alarm cleanup. 

Accessing Patch Monkey

Anyone with a NetID can access Patch Monkey:  https://patchmonkey.oit.duke.edu/

Within Patch Monkey, on the calendar view the 'red text' patch windows are for test and dev hosts.   The 'green text' patch windows are for production hosts. 

To view a list of hosts or search for a particular host select 'Computers' from the menu.

Patch Windows

• Every system that is in OIT's CMDB and marked as 'Active' with a sysadmin group of 'SSI-UNIX' or 'SSI-Windows' is in Patch Monkey, but may or may not have a patch window.
• Every VM created by clockworks that is not self administered will have a patch window automatically assigned. Sysadmins have the ability to unassign patch windows in special cases.
• Physical systems will need to be manually scheduled for a patch window after they've been appropriately added to the CMDB. 
• Every system in patch monkey that is assigned to be patched will be patched every two weeks.

Patch Window Email Notifications

There will be two separate emails sent to members of the application support group for each patch window that contains hosts that their group is listed as supporting.  One email will be sent 24 hours prior to the patch window and another email shortly after the end of the patch window. Members of the application support group will only receive an email if a server listed for their group is in a particular patch window.  If multiple hosts are patched in the same window the emails to the application support group will contain a listing of the all the hosts for that group within that window; that way the application support group will only receive two emails for each patch window regardless of the number of hosts patched.

The emails will be from patchmonkey@duke.edu.

OIT uses our CMDB to match an application support group with each host.   The application support groups are based on the ServiceNow group membership.  So anyone who is a member of the application support groups' ServiceNow group will receive the notification emails.

The email after the patching window will show one of three statuses for each host:

• Patched:  the host was successfully patched
• No Patches:  the host did not have any patches that needed to be installed
• Error:  either the attempt to patch or the reboot failed and the OIT operating system support teams have been notified to investigate the issue

Why did my server reboot when the status is ‘No Patches’?

There are two possible reasons:

• If your server is using reboot dependencies, the patching of another server may have forced the reboot of this server
• Windows servers can be placed in a ‘Reboot Required’ state after software is installed.  If this state is detected by Patch Monkey it will immediately reboot the server.

One Time Request To Not Patch A Host

If you would like to make a 'one time' request that your host not be patched during a particular patch window the request needs to be approved by the University IT Security Office (ITSO). To make the request email the following information to security@duke.edu:

• host names
• the date and time that the patch window is scheduled to start
• the reason that you are requesting that the patch window be skipped

If approved the host will not be patched in that patch window and will have the patches applied in the next regularly scheduled patch window.

Emergency Patching

Emergency patching occurs if there has been a security vulnerability that the University IT Security Office (ITSO) has determined is critical enough to warrant immediate attention. The OIT OS teams will coordinate the patching and notifications will be sent out.

Questions About Patch Monkey

Email patchmonkey@duke.edu

Network Security Scanning

The University IT Security Office scans all public IP addresses monthly looking for vulnerabilities present in OIT hosts.  Any vulnerabilities found will generate a ticket requesting that the issues are remediated.  The following are basic guidelines from the University IT Security Office to help ensure that your host is protected:

1. Always use strong, complex passwordsor passphrases for every account (especially for service accounts) and change them on a regular basis.
2. Ensure that hosts recieve security updates and patches regularly.
3. Use a host based firewall configured for default deny. This is especially important for administrative services such as SSH, RDP, and administrative access to a web application.

A full list of Server Security Standards can be found here.

New Firewall Rule Requests

Requests for the implementation of new firewall rules can be entered via a form in Cartographer.  Following the link below and filling out the request will create a ticket for the Security team to review the request and for the Networking team to implement the change.  

 https://cartographer.oit.duke.edu/firewall/requests/

Transferring Ownership, Access and Management of VMs

• In the event that the System Administrators (users who are able to access VMs with elevated administrative privileges) need to be added and/or removed then a ServiceNow task should be opened with the Systems-Vmware Storage and Backups-OIT team.  Updates to the System Administrator list will need the approval of the current Owner of the VM (most VMs in Clockworks have an associated ServiceNow group and the manager of that group is considered the owner of the VM.  In the event that there is no associated ServiceNow group, the owner is considered the primary contact listed).
• In the event that the ownership of a VM needs to be transferred then a ServiceNow task should be opened with the Systems-Vmware Storage and Backups-OIT team.  Transfers of ownership will require the approval of the new and former owners of the VM.
• In the event that a VM needs to be moved to a new network a ServiceNow task should be opened with the Systems-UNIX-OIT team or Systems-Windows-OIT team.  If the host is self-managed then a ServiceNow task should be opened with the Systems-Vmware Storage and Backups-OIT team.
• If a VM is transitioning from being Self-managed to OIT-managed then the standard procedure involves the rebuilding of the VM to meet with OIT standards.  A ServiceNow task should be opened with the Systems-UNIX-OIT team or Systems-Windows-team.
• If a VM is transitioning from being OIT-managed to Self-managed then a ServiceNow task should be opened with the Systems-UNIX-OIT team or Systems-Windows-OIT team.