Brian Parsons

Harvard Business Review has published an essay about the "sandwich approach" to giving negative feedback by Roger Schwarz. The essay takes issue with the practice of providing positive and negative feedback together. While I wouldn't say that I stick to the positive-negative-positive format of the sandwich, I do try to balance feedback when I need to address an issue with a co-worker or employee. I try to find something positive to say that is in proportion to the negative points that I have to get across. Something light for a small issue, or larger praise to accompany the discussion and resolution of a larger issue.

The article states that this waters down and undermines the message, and I can understand that. Sometimes people only hear the good things and ignore the "room for improvement".

The method that the article encourages is a "mutual learning" approach that encourages the employee to particiate in creating the structure of the discussion and the resolution:

I want to start by describing what I saw that raised my concerns and see if you saw the same things. After we agree on what happened, I want to say more about my concerns and see if you share them.

The "sandwich" approach is painted as "unilateraly controlling". I agree with that terminology in situations where the negative feedback and resolution is laid out by the manager. Typically when I address an issue, I bring collaboration into the resolution, my philosophy being that as a manager I am there to help them understand why it needs to be fixed and how to fix it.

Managers do make other mistakes with the sandwich approach. They may not keep the positive in league with the negative in terms of tone and relevant size, and they may be uncomfortable with the negative part of the conversation to the point where they minimize and gloss over it, instead of ensuring that the person heard and understood the feedback. They also may not work out a resolution and set expectations of how to measure the improvement.

In defense of my balance method I would offer an additional point. Many roles in the teams that I work with in the technology field are creative ones. Getting the most out of creative people means leading them through inspiration and not just task driving. For those people, providing negative feedback in a direct manner is quite a buzz kill and I have seen them take a few days to get their stride back.

Another issue that I have with the proposed "mutual learning" approach is that when explaining an issue or observation, asking if the person agrees puts them on the defensive. Hearing excuses then escalates my assertiveness. While I am sure that some HR policies create a rigid framework for reviews, and that there may be bad managers out there that are dictators, situations like this should never be one-way discussions. At every step the person receiving the feedback should be able to make their points, and I would frame this as a conversation and open dialog.

Overall the message of the article is something I am very in tune with - being transparent as a manager. In these kind of situations, making the process transparent and not masking the feedback in either dimished or amplified persepective is the key to improvement. It also helps the person being managed learn and grow themselves.

Update August 21, 2016

I am no longer maintaining Arch Linux images for Amazon EC2, and I no longer recommend using Arch Linux on servers. The attitude in some of the core pieces of the system has become far less disciplined and... what I will in a politically correct way say is more centered around agenda than users or system use.

Specifically the issue that broke this for me is the way versions of pacman since the file reorganization effort remove symlinks in the root path install path of a package. This bug has been brought up several times in pacman's history. The author and current Arch czar has stated that symlinks are improper and should be replaced with bind mounts. This approach breaks the best practice of being able to separate the OS from the data, and using bind mounts causes disk metrics, analysis and monitoring to misreport. In previous instances, this bug was fixed, however so far this time it is not being addressed.

I continue to be a proponent of Arch Linux for desktop use, but I have stopped using it on servers. I'm currently deploying using CentOS and most of the scripts I have open sourced for system management have been updated to work with CentOS.

Below is for Historical Purpose only.

These Arch Linux images for Amazon EC2 use my ec2-init script which requires python2 and boto, but other than that they are stock Arch Linux with just the base load and LTS kernel.

Usage Notes:

The ec2-init script will find the following variables in the user-metadata for the instance:

• hostname - The hostname to set for the instance
• mailto - the address to email with a message listing the instance information and ip address
• mailfrom - the from address of the email message

The user-metadata should be pipe delimited like this:

hostname=myhost.example.com|mailto=myemail@example.com|mailfrom=ec2host@example.com

Additionally if the instance is granted IAM role permission to Route53, the script will create or update a DNS entry for the hostname if it finds a matching zone in Route53.

Pacman is functional but key signing has not been initialized. I recommend you install haveged and initialize the package signing:

# pacman-key --init

# pacman-key --populate

The pacman-key --init command will take a while or seem like it is hung while the system gathers enough entropy for the random number generator. To help it out, you can log into another session and do an ls -lR / as it uses system activity.

See Pacman-key on the Arch Linux Wiki for more information.

I have an updated Arch Linux image for Amazon EC2 that is systemd. I created a boot script that sets the hostname and root keys. It will even update DNS in Route53 and send you an email letting you know the instance IP.

Released under the MIT license on github.

I am working on cleaning up the base image that I use on Amazon EC2 and publishing the AMI as well.

I've been running Arch Linux on my workstations and on servers for a long time. Every once in a while I see a debate in an Arch Linux forum about it's suitability for use on production servers. Being a rolling release distribution, it is different than other distributions that concentrate on enterprise and long-term support like RedHat Enterprise and CentOS. Without getting too much into the pros and cons - one of the key reasons that I use Arch on servers is earlier access to newer technologies like the 3.0 Linux kernel series (with built-in xen support). Overall, though, it is due to my familiarity with and love for it. The OS that I load on my servers is there to support my applications. I find Arch is simple and light yet thorough and stable in getting the job done. If you are running Arch on servers or are interested in doing so, here are some practices that I recommend.

This is not meant to be an exhaustive list and there are different approaches to systems administration. I welcome feedback and discussion of these concepts. I have seen projects centered around creating an off-shoot of Arch to use on servers. Ultimately I think they miss the point. The idea is not to make Arch act like CentOS. With some simple tweaks to your deployment and management process, Arch is a fine distro to use on servers.

Dealing with "Rolling Release"

Many of these things apply not just to Arch, but any rolling release distribution. Recently, Arch Linux has gone through some fundamental changes in the base layers of the operating system like the network configuration and system initialization. Updating needs to be a regular and intelligent process. You can automate much of it but you really should do the base updates manually.

1. Have a server in each datacenter or cloud that acts as a "base" server for testing updates. Always have a server that you can test updates on. I use Linode, Rackspace, and Amazon EC2 clouds and I have dev servers in each of those that are there to test updates on so that I can work out any issues before updating mission-critical instances. Once you update the base server, you can image it out appropriately for your environment.

2. Keep Snapshots so that you can "roll back". This is one of those things you should be doing no matter what you are using.

3. Update often. I run updates weekly on my workstations and weekly or bi-weekly on my servers. With a rolling release distribution, the more out of date you are, the more work you have to do with each update. If you don't have a proper environment for testing updates, make an image of your server and run updates against that in something like Virtualbox.

4. Watch the News/Forums/Mailing Lists for Update Issues. I update my workstations first and run that for a few days before updating my servers - unless it is a critical security issue. Package updates fixing security issues should always be done as soon as is practically possible.

5. Don't Run Updates via Daemon or Cron. I do not recommend running system updates via cron. You just never know when an update will require more than just the basics. If you are pushing your own applications via custom repositories, those can be automated if appropriate. (See Custom Repositories below)

6. Script tricky updates. A quick bash script can make updates against servers very simple and painless. I typically run my updates from one spot. Configuration management tools can help here too. (See Configuration Management below)

7. Remove pacman from SyncFirst and HoldPkg in /etc/pacman.conf. The default pacman.conf will stop and prompt to update pacman if there is a new version of it before updating the system. For workstations this is fine, but for servers or when you are running scripted updates, this will get in the way. If you are updating your workstations first and the server last, you will know if you need to update pacman first.

8. Create scripts to bring machines current from your vendor's image. Ideally you are running your own images made from your own base instances, but if you are using the vendor's images - such as the "Arch Linux" installs from Rackspace or Linode, you should have a script that takes that image and brings it current. This script needs to be tested and updated regularly as part of your update cycle.

Understanding the Philosophy

The key to running any operating system on your servers is understanding it and the philosophy behind it. Arch Linux is a lot like python. The driving philosophy is simplicity. Many times people over-think or assume that a given task is harder than it is.

9. Run Arch as your daily driver. Nothing will bring your knowledge up like interacting with the system on a daily basis. Linux is a great deskop OS, give Arch a try.

Also see The Arch Way entry in the Arch Linux wiki.

Custom Repositories

10. Keep private repositories in their own conf file. Instead of the main pacman.conf file, you can create your own configuration file and call pacman with --config filename. This allows you to update the packages from your repo independently of system updates.

11. SSL and ACL protect private repos. I put my custom repos behind a classic ACL username and password. Yes this is present in the .conf file URL in plaintext, but I can always change the ACL if I find it gets compromised. Using SSL will protect it in transit. Of course this isn't foolproof so if you are concerned about your proprietary packages leaking, watch the logs or load the sensitive packages in a different way.

12. Sign your packages. Arch's package management system now supports package signing and verification.

13. Keep workstation and server repos separated. I build custom packages that I use on workstations and servers. I like to keep them in different repos so that the server repo stays as clean as possible.

Configuration Management

14. Configuration Management is your friend. If you are managing multiple servers, configuration management tools like Puppet, Chef, or CFEngine are your friend. Employing one of these tools properly will keep your servers consistent and greatly ease management and deployment.

Again, these practices are not meant to be all-encompassing. There are probably many other things that could go into this, but I hope sharing my approach can help others. The Arch Linux Wiki, Forums, and IRC Channels are always helpful resources.

The book, The CIO Paradox, by Martha Heller identifes some common contradicitions of managing technology, grouped into four main categories:

Your Role:

• You were hired to be strategic, but you are forced to spend most of your time on operational issues.
• You are the steward of risk mitigation and cost containment, yet you are expected to innovate.
• Your function is seen as that of an enabler, yet you are also expected to be a business driver.
• IT can make or break a company, but you are not a member of the corporate board.

Your Stakeholders:

• You run one of the most pervasive, critical functions, yet you must prove your value constantly.
• Your many successes are invisible; your few mistakes are highly visible.
• You are intimately involved in every facet of the business, yet you are considered separate and removed from it.
• You are accountable for project success, but the business has ownership.

Your Organization/Team:

• Your staff loves technology but must embrace business to advance.
• Your team members are uncomfortable with people, but to succeed they must build relationships and influence others.
• You develop successors, yet the CEO almost always goes outside for the next CIO.
• You are forced to seek less expensive overseas sourcing, yet you are expected to ensure the profession.

Your Industry:

• Technology takes a long time to implement, yet your tool set changes constantly.
• Technology is a long-term investment, but the company thinks in quarters.
• Your tools cost a fortune, yet have the highest defect rate of any product.
• You sign vendors' checks, yet they try their darndest to sell to your business peers.

I could add another category, "Your Management". In many organizations, technology reports into finance. This was a natural fit because finance and accounting are in many cases the biggest "users" of IT and IT is seen as a cost center that needs to be reigned in. The book makes the case for the new IT paradigm which is based on visibility and accountability. Many organizations evolve slowly, however so those challenges are still real. According to a recent study by Gartner, the CMO will grow to be the biggest IT spender by 2017. Does this mean IT will eventually report to Marketing?

Acquisitions are a major component of growing businesses, and often you see managers being put over product lines or services that they have no expertise in. Whether it is accounting/CFO, marketing/CMO or operations/COO, many senior managers charged with managing the technology team don't understand technology or how to manage it. The paradoxes that this creates for technology managers are:

• Management makes decisions and promises based on ego instead of fact, often blaming technology when the facts don't line up with their assumptions.
• You are told to "own" the roadmap, uptime, and technology behind the product or service, yet you are micromanaged to failure.
• Management has strategy meetings but doesn't include technology leadership or see technology as an important partner.
• Your management constantly shifts direction and requirements away from commitments made, leaving technology saddled with the image of not delivering or responding quickly enough.

Like the other CIO paradoxes, most of these are addressable or are symptoms of a different problem such as corporate structure or even bad management.

I highly recommend Martha Heller's book as good reading for any technology manager or even developers who face some of these same challenges. Solving many of the "paradoxes" comes down to communicating effectively and creating the right kind of visibility. The tactics and approaches illustrated in the book are helpful in any (functional) organization.

If you are stuck with a bad manager, you can always offer them some relevant reading.