Pythian Blog: Technical Track

SaltStack for remote parallel execution of commands

by Chris Stankaitis on Jul 7, 2014 12:00:00 AM

There are many scenarios when a SysAdmin has to do a "box walk" of the entire infrastructure to execute a command across many servers. This is universally accepted as one of the less glamorous parts of our job. The larger the infrastructure, the longer these box walks take, and the greater chance that human error will occur. Even giving this task to a junior resource, as is often the case, is not sustainable as the infrastructure grows, and does not represent the best value to the business in terms of resource utilization. Additionally, too much of this type of "grind" work can demoralize even the most enthusiastic team member. Thankfully the days of having to do these box walks are over. Thanks to configuration management and infrastructure automation tools, the task has been automated and no longer requires the investment in time by a human SysAdmin that it once did. These tools allow you, at a very high level, to off load this repetitive work to the computer, with the computer doing the heavy lifting for you.  

Introducing SaltStack

SaltStack is a distributed remote execution system used to execute commands and query data on remote nodes, either individually or by arbitrary selection criteria. SaltStack is also a configuration management system in it's own right but this post will be focusing on Salt from a "Command and Control" point of view. Salt has 2 main components, the "salt master" (server) and the "salt minions" (clients). Once the minions are accepted by the master, then further execution of commands can come directly from the central salt master server. Once you have installed your packages the minion needs to be configured to know where its master is. This can be accomplished through a DNS or hosts-file entry or by setting the variable in the /etc/salt/minion config. [code language="bash"] master: XXX.XXX.XXX.XXX [/code] Where "XXX.XXX.XXX.XXX" is the IP Address of your master server. Once that is done, and the salt-minion service has been started the minion will generate and ship an SSL key back to the master to ensure all communication is secure. The master must accept the key from the minion before any control can begin. [code language="bash"] # Listing the Keys [root@ip-10-154-193-216 ~]# salt-key -L Accepted Keys: Unaccepted Keys: ip-10-136-76-163.ec2.internal Rejected Keys: # Adding The Key [root@ip-10-154-193-216 ~]# salt-key -A The following keys are going to be accepted: Unaccepted Keys: ip-10-136-76-163.ec2.internal Proceed? [n/Y] y Key for minion ip-10-136-76-163.ec2.internal accepted. # Nailed It! Now the Master can control the Minion! [root@ip-10-154-193-216 ~]# salt-key -L Accepted Keys: ip-10-136-76-163.ec2.internal Unaccepted Keys: Rejected Keys: [/code] Note: Not Shown - I added a 2nd Minion Now that your master has minions the fun begins. From your master you can now query information from your minions such as disk space: [code language="bash"] [root@ip-10-154-193-216 ~]# salt '*' disk.percent ip-10-136-76-163.ec2.internal: ---------- /: 15% /dev/shm: 0% ip-10-147-240-208.ec2.internal: ---------- /: 14% /dev/shm: 0% [/code] And you can also execute remote commands such as finding out service status, and restarting services. [code language="bash"] [root@ip-10-154-193-216 ~]# salt '*' cmd.run "service crond status" ip-10-136-76-163.ec2.internal: crond (pid 1440) is running... ip-10-147-240-208.ec2.internal: crond (pid 1198) is running... [root@ip-10-154-193-216 ~]# salt '*' cmd.run "service crond restart" ip-10-136-76-163.ec2.internal: Stopping crond: [ OK ] Starting crond: [ OK ] ip-10-147-240-208.ec2.internal: Stopping crond: [ OK ] Starting crond: [ OK ] [/code] These are only the most basic use cases for what Salt Stack can do, but even from these examples it is clear that salt can become a powerful tool which can reduce the potential for human error and increase the efficiency of your SysAdmin Team. By Implementing Configuration Management and Infrastructure Automation tools such as Salt Stack you can free up the time of your team members to work on higher quality work which delivers more business value. Salt Stack (depending on your setup) can be deployed in minutes. On RHEL/CentOS/Amazon Linux using the EPEL repo I was able to be up and running with Salt in about 5 minute on the 3 nodes I used for the examples in this post. Salt can be deployed using another configuration management tool, it can be baked into your provisioning environment, or into base images. If all else fails, (ironically) you can do a box walk to install the package on your existing servers. Even if you have another configuration management solution deployed, depending on what you are trying to accomplish using Salt for parallel command execution rather then the Config Management system can often prove a much simpler and lightweight solution. Salt is also a great choice in tools for giving other teams access to execute commands on a subset of boxes without requiring them to have shell access to all of the servers. This allows those teams to get their job done without the SysAdmin team becoming a bottle neck.

No Comments Yet

Let us know what you think

See all

See all

Subscribe by email
Follow us
Love Your Data
Pythian Logo

© 2024 All rights reserved.