Brian Parsons

If you are going to the 2017 NAB Radio show in Austin, please come and see the session that I've put together with some help from my Vipology co-horts, "Unlock Higher Ratings and Revenue Through Data".

I'm really glad to have been engaged by the NAB to help them with content again this year. So that I don't have to talk for 45 minutes - and you don't want to hear it all from me anyway - we've assembled a panel of high-level experts when it comes to the impact that technology has had on research and media buying.

The session is Thursday, September 7th on the Innovation Stage. More info here: http://www.radioshowweb.com/sessions/sessiondetail.asp?id=1211325

Cheers,

I've been reading about slow growth for Rackspace cloud and how apps are pulling support. I shouldn't be so surprised given that my own usage of the Rackspace cloud has also dwindled, despite the ORD datacenter being one of the most rock solid facilities I have ever used. I know that Rackspace has spent the last few years working hard and innovating, but somehow they seem to still be missing the boat. Here is a list of key things that made me go back to AWS and that Rackspace can implement to reverse this trend.

1. Bring Your Own Kernel/Image - One of the big issues I have with Rackspace is flexibility. This is also an area where AWS shines. When it comes to vm images, AWS has built a marketplace around them. There are thousands. Rackspace has a 27 starter images in their "First Generation" platform and 49 images in their "2nd Generation".

2. Get rid of storage templates - Rackspace's new Open Stack storage lets you add block storage from 100GB-1024GB in size. On AWS I can allocate pretty much any size I want.

3. Notify Customers When Logging Into an Instance - One of the eerie things about using Rackspace cloud is that I randomly get logins from Rackspace staff. They have a few daemons they set up and like to run on the instance. Since I use Arch Linux and I keep it up to date, their daemon sometimes gets turned off due to dependencies it has getting upgraded (they don't use a package!). So every once in a while I log into an instance and see someone's been there, tinkering with the daemon and getting it to run. I wouldn't mind this if there was communication. Or maybe I would. I would rather get a ticket "Please run this software on your instances or ask us if you'd like us to install it."

4. Add an API for Health Checks/Failover - Provide basic scripting with the monitoring so that failover can be initiated. (Short cut: Buy a DNS provider like Ultra or Dyn)

5. Mirror Databases in Both Datacenters - Let me configure MySQL replication between DFW and ORD.

Being able to copy cloudfiles and machine images between the two datacenters (which AWS introduced a few months back) is very handy for building resilient distributed infrastructure as well, but I think that the list of five items above would go a long way to stem the bleeding. Since Rackspace does have more memory options, it could prove to be the more flexible platform. While I'm at it though, three more wish list items:

6. Unify the US and UK Control Panels - Having to log in and manage Rackspace UK separately is a pain.

7. Multi-Factor Authentication for the Console - there are many ways to get MFA into your infrastructure but the management console is still MFA-less. I have read that Rackspace is working on this, and they may very well be working on other items on this list.

8. Encrypted Server Images - Since there is console support you could completely secure an instance. This isn't practical for every application. I encrypt a lot of my storage at Amazon, but since AWS doesn't have interactive console support, the boot chain remains unencrypted. This is another way that Rackspace could pull ahead, although much like the templating on OS and block storage it seems like Rackspace is really tied to the philosophy of managed services instead of manage your own. Encrypting the images could become a hurdle to upgrading to managed services.

The pace of innovation coming out of Amazon is really amazing for a company that size. I know many hard working people at Rackspace as well and they aren't standing still. Maybe the difference is a little marketing, but a few of the items on my list point to a shift in perspective on the Rackspace cloud product offering itself. An "open" cloud behind a set of pre-defined templates doesn't seem to be all that flexible or fun.

Harvard Business Review has published an essay about the "sandwich approach" to giving negative feedback by Roger Schwarz. The essay takes issue with the practice of providing positive and negative feedback together. While I wouldn't say that I stick to the positive-negative-positive format of the sandwich, I do try to balance feedback when I need to address an issue with a co-worker or employee. I try to find something positive to say that is in proportion to the negative points that I have to get across. Something light for a small issue, or larger praise to accompany the discussion and resolution of a larger issue.

The article states that this waters down and undermines the message, and I can understand that. Sometimes people only hear the good things and ignore the "room for improvement".

The method that the article encourages is a "mutual learning" approach that encourages the employee to particiate in creating the structure of the discussion and the resolution:

I want to start by describing what I saw that raised my concerns and see if you saw the same things. After we agree on what happened, I want to say more about my concerns and see if you share them.

The "sandwich" approach is painted as "unilateraly controlling". I agree with that terminology in situations where the negative feedback and resolution is laid out by the manager. Typically when I address an issue, I bring collaboration into the resolution, my philosophy being that as a manager I am there to help them understand why it needs to be fixed and how to fix it.

Managers do make other mistakes with the sandwich approach. They may not keep the positive in league with the negative in terms of tone and relevant size, and they may be uncomfortable with the negative part of the conversation to the point where they minimize and gloss over it, instead of ensuring that the person heard and understood the feedback. They also may not work out a resolution and set expectations of how to measure the improvement.

In defense of my balance method I would offer an additional point. Many roles in the teams that I work with in the technology field are creative ones. Getting the most out of creative people means leading them through inspiration and not just task driving. For those people, providing negative feedback in a direct manner is quite a buzz kill and I have seen them take a few days to get their stride back.

Another issue that I have with the proposed "mutual learning" approach is that when explaining an issue or observation, asking if the person agrees puts them on the defensive. Hearing excuses then escalates my assertiveness. While I am sure that some HR policies create a rigid framework for reviews, and that there may be bad managers out there that are dictators, situations like this should never be one-way discussions. At every step the person receiving the feedback should be able to make their points, and I would frame this as a conversation and open dialog.

Overall the message of the article is something I am very in tune with - being transparent as a manager. In these kind of situations, making the process transparent and not masking the feedback in either dimished or amplified persepective is the key to improvement. It also helps the person being managed learn and grow themselves.

I've been running Arch Linux on my workstations and on servers for a long time. Every once in a while I see a debate in an Arch Linux forum about it's suitability for use on production servers. Being a rolling release distribution, it is different than other distributions that concentrate on enterprise and long-term support like RedHat Enterprise and CentOS. Without getting too much into the pros and cons - one of the key reasons that I use Arch on servers is earlier access to newer technologies like the 3.0 Linux kernel series (with built-in xen support). Overall, though, it is due to my familiarity with and love for it. The OS that I load on my servers is there to support my applications. I find Arch is simple and light yet thorough and stable in getting the job done. If you are running Arch on servers or are interested in doing so, here are some practices that I recommend.

This is not meant to be an exhaustive list and there are different approaches to systems administration. I welcome feedback and discussion of these concepts. I have seen projects centered around creating an off-shoot of Arch to use on servers. Ultimately I think they miss the point. The idea is not to make Arch act like CentOS. With some simple tweaks to your deployment and management process, Arch is a fine distro to use on servers.

Dealing with "Rolling Release"

Many of these things apply not just to Arch, but any rolling release distribution. Recently, Arch Linux has gone through some fundamental changes in the base layers of the operating system like the network configuration and system initialization. Updating needs to be a regular and intelligent process. You can automate much of it but you really should do the base updates manually.

1. Have a server in each datacenter or cloud that acts as a "base" server for testing updates. Always have a server that you can test updates on. I use Linode, Rackspace, and Amazon EC2 clouds and I have dev servers in each of those that are there to test updates on so that I can work out any issues before updating mission-critical instances. Once you update the base server, you can image it out appropriately for your environment.

2. Keep Snapshots so that you can "roll back". This is one of those things you should be doing no matter what you are using.

3. Update often. I run updates weekly on my workstations and weekly or bi-weekly on my servers. With a rolling release distribution, the more out of date you are, the more work you have to do with each update. If you don't have a proper environment for testing updates, make an image of your server and run updates against that in something like Virtualbox.

4. Watch the News/Forums/Mailing Lists for Update Issues. I update my workstations first and run that for a few days before updating my servers - unless it is a critical security issue. Package updates fixing security issues should always be done as soon as is practically possible.

5. Don't Run Updates via Daemon or Cron. I do not recommend running system updates via cron. You just never know when an update will require more than just the basics. If you are pushing your own applications via custom repositories, those can be automated if appropriate. (See Custom Repositories below)

6. Script tricky updates. A quick bash script can make updates against servers very simple and painless. I typically run my updates from one spot. Configuration management tools can help here too. (See Configuration Management below)

7. Remove pacman from SyncFirst and HoldPkg in /etc/pacman.conf. The default pacman.conf will stop and prompt to update pacman if there is a new version of it before updating the system. For workstations this is fine, but for servers or when you are running scripted updates, this will get in the way. If you are updating your workstations first and the server last, you will know if you need to update pacman first.

8. Create scripts to bring machines current from your vendor's image. Ideally you are running your own images made from your own base instances, but if you are using the vendor's images - such as the "Arch Linux" installs from Rackspace or Linode, you should have a script that takes that image and brings it current. This script needs to be tested and updated regularly as part of your update cycle.

Understanding the Philosophy

The key to running any operating system on your servers is understanding it and the philosophy behind it. Arch Linux is a lot like python. The driving philosophy is simplicity. Many times people over-think or assume that a given task is harder than it is.

9. Run Arch as your daily driver. Nothing will bring your knowledge up like interacting with the system on a daily basis. Linux is a great deskop OS, give Arch a try.

Also see The Arch Way entry in the Arch Linux wiki.

Custom Repositories

10. Keep private repositories in their own conf file. Instead of the main pacman.conf file, you can create your own configuration file and call pacman with --config filename. This allows you to update the packages from your repo independently of system updates.

11. SSL and ACL protect private repos. I put my custom repos behind a classic ACL username and password. Yes this is present in the .conf file URL in plaintext, but I can always change the ACL if I find it gets compromised. Using SSL will protect it in transit. Of course this isn't foolproof so if you are concerned about your proprietary packages leaking, watch the logs or load the sensitive packages in a different way.

12. Sign your packages. Arch's package management system now supports package signing and verification.

13. Keep workstation and server repos separated. I build custom packages that I use on workstations and servers. I like to keep them in different repos so that the server repo stays as clean as possible.

Configuration Management

14. Configuration Management is your friend. If you are managing multiple servers, configuration management tools like Puppet, Chef, or CFEngine are your friend. Employing one of these tools properly will keep your servers consistent and greatly ease management and deployment.

Again, these practices are not meant to be all-encompassing. There are probably many other things that could go into this, but I hope sharing my approach can help others. The Arch Linux Wiki, Forums, and IRC Channels are always helpful resources.

The book, The CIO Paradox, by Martha Heller identifes some common contradicitions of managing technology, grouped into four main categories:

Your Role:

• You were hired to be strategic, but you are forced to spend most of your time on operational issues.
• You are the steward of risk mitigation and cost containment, yet you are expected to innovate.
• Your function is seen as that of an enabler, yet you are also expected to be a business driver.
• IT can make or break a company, but you are not a member of the corporate board.

Your Stakeholders:

• You run one of the most pervasive, critical functions, yet you must prove your value constantly.
• Your many successes are invisible; your few mistakes are highly visible.
• You are intimately involved in every facet of the business, yet you are considered separate and removed from it.
• You are accountable for project success, but the business has ownership.

Your Organization/Team:

• Your staff loves technology but must embrace business to advance.
• Your team members are uncomfortable with people, but to succeed they must build relationships and influence others.
• You develop successors, yet the CEO almost always goes outside for the next CIO.
• You are forced to seek less expensive overseas sourcing, yet you are expected to ensure the profession.

Your Industry:

• Technology takes a long time to implement, yet your tool set changes constantly.
• Technology is a long-term investment, but the company thinks in quarters.
• Your tools cost a fortune, yet have the highest defect rate of any product.
• You sign vendors' checks, yet they try their darndest to sell to your business peers.

I could add another category, "Your Management". In many organizations, technology reports into finance. This was a natural fit because finance and accounting are in many cases the biggest "users" of IT and IT is seen as a cost center that needs to be reigned in. The book makes the case for the new IT paradigm which is based on visibility and accountability. Many organizations evolve slowly, however so those challenges are still real. According to a recent study by Gartner, the CMO will grow to be the biggest IT spender by 2017. Does this mean IT will eventually report to Marketing?

Acquisitions are a major component of growing businesses, and often you see managers being put over product lines or services that they have no expertise in. Whether it is accounting/CFO, marketing/CMO or operations/COO, many senior managers charged with managing the technology team don't understand technology or how to manage it. The paradoxes that this creates for technology managers are:

• Management makes decisions and promises based on ego instead of fact, often blaming technology when the facts don't line up with their assumptions.
• You are told to "own" the roadmap, uptime, and technology behind the product or service, yet you are micromanaged to failure.
• Management has strategy meetings but doesn't include technology leadership or see technology as an important partner.
• Your management constantly shifts direction and requirements away from commitments made, leaving technology saddled with the image of not delivering or responding quickly enough.

Like the other CIO paradoxes, most of these are addressable or are symptoms of a different problem such as corporate structure or even bad management.

I highly recommend Martha Heller's book as good reading for any technology manager or even developers who face some of these same challenges. Solving many of the "paradoxes" comes down to communicating effectively and creating the right kind of visibility. The tactics and approaches illustrated in the book are helpful in any (functional) organization.

If you are stuck with a bad manager, you can always offer them some relevant reading.