RE: [Puppet Users] RE: enterprise puppet architecture

Your Puppet master has twice the CPU of ours; but more importantly, you have 
far simpler manifests.  Ours are very complex, and can take 20seconds on 
average to build - some taking a minute for the whole process to finish.

We're going to completely redesign our setup, as per the instructionsin the 
ProPuppet book, with multiple puppetmasters in a cluster behind a load balancer 
so that we can expand indefinitely.

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Ph: +64 9 373 7599 ext 86487

________________________________
From: [EMAIL PROTECTED] [EMAIL PROTECTED] on behalf of Luke Bigum [EMAIL 
PROTECTED]
Sent: Thursday, 23 February 2012 9:56 p.m.
To: [EMAIL PROTECTED]
Subject: Re: [Puppet Users] RE: enterprise puppet architecture

On 23/02/12 07:26, Steve Shipway wrote:
Our Puppet system here is currently managing about 500 nodes.  We anticipate 
about 1000 eventually.

I have had to reduce the client frequency to once every 4 hours; it seems that 
the maximum that can be handled by a single (dual-CPU, 8GB) puppet master is 
200 nodes.  After that, performance drops quickly and I notice many failed 
manifests.  This is with Puppet 2.7.10 on the master.


Hi Steve,

Excuse the slight change in topic but I'm interested in the performance stats 
you posted. I run Puppet 2.7.5 on a 4 CPU 4 GiB RAM KVM virtual machine. I use 
Puppet Commander to evenly distribute runs and my interval time works out to be 
around 15 minutes for 230 odd hosts, as per the timestamps between MCollective 
discoveries below:

[EMAIL PROTECTED] ~]# grep Found /var/log/puppetcommander.log | tail
I, [2012-02-23T06:46:12.218853 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T06:57:59.009689 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T07:09:49.237810 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T07:21:39.435558 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T07:33:26.554525 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T07:45:59.550541 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T07:57:51.013245 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T08:12:10.915308 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T08:24:16.383794 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs
I, [2012-02-23T08:37:03.750438 #28284]  INFO -- : Found 231 puppet nodes, 
sleeping for ~3 seconds between runs

I allow 10 Agents to run concurrently however my catalogs are very very light, 
less than a second to compile:

[EMAIL PROTECTED] ~]# grep 'Compiled catalog' /var/log/messages | awk 
'{sum+=$14} END {print sum/NR}'
0.750115

How big are your Puppet manifests so that you've had to drop the run time down 
to 4 hours? Have you considered the use of MCollective and Puppet Commander to 
spread your load out more?

-Luke

We've bought a copy of ProPuppet (as  Jeff Watts recommended) and we're 
planning to make a distributed system as instructed in there -- one puppet 
dashboard/report server, multiple puppet master servers, and one dev server.  
Puppet configurations held is subversion and synchronised on all puppet 
masters, which would themselves be behind a load balancer.  This is still in 
the planning stage, though.

I'd be interested in hearing your experiences in managing your extra-large 
system; I can also share our experiences in how we implemented and manage 
control of this system, if you'd like to contact me off-list.  When we first 
implemented, we engaged a Puppet Labs consultant for a few days to help with 
the initial work.  I can definitely recommend doing this if you've no puppet 
experience, as one place Puppet lacks is documentation!

Steve

Steve Shipway
University of Auckland ITS
UNIX Systems Design Lead
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Ph: +64 9 373 7599 ext 86487

--
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to [EMAIL PROTECTED]<mailto:[EMAIL 
PROTECTED]>.
To unsubscribe from this group, send email to [EMAIL PROTECTED]<mailto:[EMAIL 
PROTECTED]>.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.



--
Luke Bigum

Information Systems
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]> | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN

The information in this e-mail and any attachment is confidential and is 
intended only for the named recipient(s). The e-mail may not be disclosed or 
used by any person other than the addressee, nor may it be copied in any way. 
If you are not a named recipient please notify the sender immediately and 
delete any copies of this message. Any unauthorized copying, disclosure or 
distribution of the material in this e-mail is strictly forbidden. Any view or 
opinions presented are solely those of the author and do not necessarily 
represent those of the company.

--
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to [EMAIL PROTECTED]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.