For Solaris 2.8

This example /etc/system is divided into four sections:

  1) CRITICAL

  2) IMPORTANT

  3) BASIC, required parameters not important to performance

  4) HISTORIC, Parameters that are no longer necessary for 2.8

******************************************************

Drop explicit variable settings if and whenever possible.
Especially when installing a new version of the OS.

Do not copy this file into /etc/system!  It is meant to be offer
suggestions on a variable-by-variable basis that are ready to
paste into with a mouse. Values entered into /etc/system are
decimal by default, hexadecimal is specified with a leading "0x".

These explanations of potential tuning values for /etc/system
can be merged ideas into your current /etc/system file.

Reference:
http://docs.sun.com/ab2/coll.707.1/SOLTUNEPARAMREF

CRITICAL --  Large System kernel parameters.

Maximum Shared memory segment size

Set according to application program requirements along with
other System V IPC parameters: shm msg and sem. This one doesn't
use any kernel resource to set high, so usually set to value >
physical mem.  See the docs.sun.com References for descriptions
of other IPC parameters.  Set to The largest shared memory
segment allowed.

DEFAULT: 1048576

UNITS: Bytes

REBOOT REQUIRED: YES

RECOMMENDATION

unlimited ...

set shmsys:shminfo_shmmax=0xffffffffffff

UFS write throttling High-Water mark:

The default for this is quite low and needs to be increased if
the system has high-speed IO devices underlying the UFS
filesystem.  The counter ufs_throttles will accumulate each time
the system throttles on this condition.  So check this and if
ufs_throttles is a large value you need to increase ufs_HW.
Recommendation, have at least 1-track, 64k * # disks in stripe,
OR at least maxphys.

DEFAULT: 393216

UNITS: Bytes of outstanding IO on a per file basis

REBOOT REQUIRED: YES

set ufs:ufs_HW=4194304

fsflush tuning

The default for these are 30 seconds for autoup, and 5 seconds
for fsflushr, This will make the fsflush process too hot on a
big memory machine.  Cool it out by reducing the cycle time,
autoup and/or increasing the run interval: tune_t_fsflushr The
system may exhibit periodic freezes when fsflush runs. To
troubleshoot this performance problem, try temporarily setting
dopageflush to 0 to turn off fsflush all together, and/or set
doiflush=0 to turn off inode flushing only.  Do this with adb
for just a few minutes!

autoup

DEFAULT: 30

UNITS: seconds

REBOOT REQUIRED: YES

set autoup=900

tune_t_fsflushr

DEFAULT: 5

UNITS: seconds

REBOOT REQUIRED: YES

set tune_t_fsflushr=1

maxphys

For 2.5.1 and higher kernels, we have the capability to do large
IOs to and through the lowest layers. This is important for
DSS.  Also "plaid" striping where veritas is used to stripe
RAID-5 Luns with the recommended 512k or 1024k Stripe
interlace.  Be sure to check your settings for volume manager
software to take complete control of the blocksize out to the IO
device. (See vxio:vol_maxio and or sd_maxphys sections.)
WARNING: this has a side-effect on maxcontig for sd devices.
See maxcontig in "NOTES" section at the end of this document.

DEFAULT: 131072

UNITS: bytes

REBOOT REQUIRED: YES

set maxphys=1048576

md_maxphys

Maximum size of IO going through the metadisk layer of DiskSuite

DEFAULT: 131072

UNITS: bytes

REBOOT REQUIRED: YES

set md_maxphys=1048576

vxio:vol_maxio

Maximum size of IO going through the VxVM layer

DEFAULT: 512, or 256KBytes

UNITS: sectors, or 512 byte units

REBOOT REQUIRED: YES

RECOMMENDATION:  Leave alone for recommened 64 or 128k stripe
interlace.  Match maxphys, for wide stripe interlace.  Useful
for hardware RAID when subsequent stripe interlace will go to
64kstripe interlace will go to 64k

set vxio:vol_maxio = 2048

*************************************************************
IMPORTANT --  Large System kernel parameters.

Soft Affinity

rechoose_interval

UNITS: Ticks

REBOOT REQUIRED: YES

This variable introduces a bias on how often the kernel moves
threads when unevenly balanced process-to-processor conditions
occur.  If a process hasn't run in rechoose_interval ticks it
will be moved to another CPU. Otherwise it will continue to wait
on the CPU it has been running on.

A higher value of rechoose_interval "firms-up" the soft
affinity. The down side is that you can end up with sluggish
spreading out of processes on an application where a single
processes forks a lots of children if this value is too high.  A
problem where threads migrate unnecessarily (Bugid 4103680) has
been fixed in 2.7.

DEFAULT: 3

RECOMMENDATION: Some applications benefit from an increase into
the range of 3 and 300.

REBOOT REQUIRED: YES

set rechoose_interval=30

smallfile -- Size file files, smaller will not be considered for
freebehind

Smallfiles are kept in cached even if sequential reads are
detected.  Large memory machines contain enough memory to cache
1000 files of 10M each without making undo memory demands. This
is highly application dependent.  The goal is to get cache
reuse, without going overboard and causing memory shortfall by
caching too much.

DEFAULT: 32768

UNITS: Bytes

REBOOT REQUIRED: No

set smallfile=10485760


bufhwm -- Buffer High Water Mark

Limits the amount of memory devoted to buffer headers cache.
The default is too high for large memory machines, set smaller
and you can recover nearly all that space.  Watch for values too
small with sar -b.

DEFAULT: 2%  of all physical memory.

UNITS: KBytes

REBOOT REQUIRED: YES

set bufhwm=8000


maxusers Many of the process-table sizes are calculated off of
maxusers.  It can be considered a "kitchen sink knob" for
dynamic sizing.  Setting this smaller for a machine that runs
database network connections that don't count as users, will
save some memory by making the kernel smaller.

DEFAULT: Approximately 1000 per GB of memory. But never less
than 8, or more than 2048.

UNITS: users

REBOOT REQUIRED: YES

set maxusers=2048

max_nprocs The maximum number of processes allowed.

DEFAULT: 16* maxusers (approx.  actually 10+16*maxusers)

UNITS: processes

REBOOT REQUIRED: YES

set max_nprocs=32768


segspt_minfree

Solaris 2.6 and above leaves at least this much memory free when
allocating ISM shared memory for parts of the kernel that
haven't paged in yet. This is because ISM shared memory implies
non-pagable memory, and thus the kernel assumes it will never
get to use this memory. The default is too high for large memory
machines and would make more sense to be limited to a few dozen
MB. Becomes CRITICAL if you are using a try to allocate all
memory for the database shared memory pool.

DEFAULT: 5% of available real memory.

UNITS: pages (8kBytes for UltraSPARC, 4k for SuperSPARC)

REBOOT REQUIRED: YES

set segspt_minfree=12000

************************************************

VxVM --  Large System kernel parameters.

see the veritas volume manager performance manual, p/n
805-1607-10

CRITICAL

************************************************

vol_maxio

IOs of a size larger than this are broken up in the Veritas VxVM
layer

DEFAULT 512 UNITS SECTORS, 4096 In vxvm 3.0.2

set vxio:vol_maxio=8192

set vxio:vol_maxkiocount=4096

vxvm 2.x:

Critical for large RAID-5 volumes.

set vxio:voliomem_max_memory=134217728

set vxio:voliomem_chunk_size=1048576

vxvm 3.x:

new 3.0.1 tunable, default 4M, set to 128M.

set vxio:voliomem_maxpool_sz=134217728

set vxio:voliomem_kvmap_size=15728640

IMPORTANT

set vxio:vol_maxioctl=131072

set vxio:vol_maxspecialio=10240

set vxio:voliomem_base_memory=104857600

This one should be at least larger than voliomem_base_memory

set vxio:voliomem_kvmap_size=135266304

USEFUL

set vxio:vol_default_iodelay=5

***********************************************

BASIC --  Large System kernel parameters.


kernel_cage_enable

Necessary to enable DR.

DEFAULT 0

UNITS: Boolean  32-bit

REBOOT REQUIRED: Yes

set kernel_cage_enable=1

rlim_fd_max

Increases the maximum number of open file descriptors. This
variable does not have a well defined maximum. Some workloads
(SpecWeb) have been run with it set very high (260000).

DEFAULT 1024

UNITS: number of files, 32-bit

REBOOT REQUIRED: YES

set rlim_fd_max=2048

rstchown

Posix Restricted chown

DEFAULT = 1 (enable restriction)

REBOOT REQUIRED: YES

set rstchown = 0


report_ce_log

DEFAULT = 1 on Starfire & Sunfire, 0 on other platforms for 2.6

DEFAULT = 0 For all machines in 2.5

REBOOT REQUIRED: NO

set report_ce_log=1

*************************************************************
HISTORIC --  Large System kernel parameters.

This sections documents few parameters that may be found as
"Legacy" setting

pageout tuning:

The recommendation is to clean out any old tuning values that
explicitly set lotsfree, minfree desfree, pages_before_pager,
fastscan or slowscan.  Under Solaris 8, a new VM algorithm has
been instituted. It is still possible to minimize buffer space
with these variables on systems with well known and fairly
static memory demands.

lotsfree

DEFAULT: 1/64 of memory, but at least 512K. (for 2.6 and 2.5.x,
same on 7)

UNITS: pages (8kBytes for UltraSPARC, 4k for SuperSPARC)

REBOOT REQUIRED: NO

To be fully compliant with paramaters as in after reboot, desfree
needs to be set to lotsfree/2 and minfree set to desfree/2.

RECOMMENDATION: If not using priority paging: Set to twice the
amount of memory the system may need to allocate in a few
seconds.  i.e. Set as high as 2 GB for parallel or batch
processing, default of 1/2 GB is OK for time-share, OLTP
server.  When using priority paging, leave this one alone.
 ... see cachefree...

set lotsfree=0x20000

priority_paging

This is the top level control for the new style virtual memory
manager policy.

DEFAULT: 0 UNITS: Boolean REBOOT REQUIRED: yes set
priority_paging=1

cachefree

This variable sets the threshold of free memory the system needs
before it begins to run fsflush. Need priority_paging set to
enable this feature.  The ideal value for cachefree will be
equal to the amount of RSS in unshared memory. The default of 2X
lotsfree should OK most of the time.

DEFAULT: 2X of lotsfree if priority_paging is set.

UNITS: pages (8kBytes for UltraSPARC, 4k for SuperSPARC)

REBOOT REQUIRED: NO

REQUIRES: set priority_paging=1

RECOMMENDATION: Leave at default of 2 * lotsfree or (1/32 of
physmem) OR Set to have the gap between lotsfree and cachefree
to be roughly the size of all executable memory needed.
Double-check with memstat.

set cachefree=0x40000

pages_before_pager tuning

DEFAULT: 200

UNITS: pages (8kBytes for UltraSPARC, 4k for SuperSPARC)

REBOOT REQUIRED: NO

This does not apply for machines using raw or direct or
cachefree kernel.  Pages_before_pager is the amount of cushion
above lotsfree where the ufs vnode cache (the big cache) gets
turned off/on for IOs using read(2)/write(2) access.  Processes
using mmap(2) ignore this and can push the system into memory
crunch more easily.  But for jobs using read(2)write(2) an
understanding of pages_before_pager may help tune out ufs/vm
problems.

The kernel will not save the vnode block if :

1. Using read(2)/write(2) AND

2. freemem is less than lotsfree + pages_before_pager  OR
Sticky-bit is set but execute bit is not set, (*traditional mode
for swap file Under 2.6 directio is set on the file or
filesystem being read.

Recommendation is to set this between 10% and 30% for machines
with RDBMS on UFS.  This sets it to 1GB for a large-memory
starfire:

set pages_before_pager=125000

set hme:hme_adv_autoneg_cap=0 IF you do this, you must turn off
autoneg here, you have to do it on the switch too.  Both sides
of the link have to be in the same mode!  set
hme:hme_adv_100hdx_cap=0 set hme:hme_adv_100fdx_cap=1

Same effect can be gotten with ndd:  ndd -set /dev/hme
adv_autoneg_cap 0 ndd -set /dev/hme adv_100hdx_cap 0 ndd -set
/dev/hme adv_100fdx_cap 1


************************************************* NOTES -- Other
tuning pointers

Priority paging is not normally necessary with Solaris 8.
Eliminate other "legacy" tuning that has been done in the
vm/pageout area as workarounds to problems that priority paging
solves. This includes specific tuning to these parameters:
lotsfree, desfree, minfree throttlefree, fastscan, slowscan,
handspreadpages.

The variables fastscan and slowscan are used when "pageout" is
running & the amount of free memory falls below lotsfree. These
variables define the number of pages examined on each run of
pageout.

In Solaris 7 and higher, the Starfire uses a custom dispatch
table loaded from /platform/sun4u1/kernel/sched/ ... This is a
binary kernel loadable module that implements a wider time slice
quanta, and a more spread- out sequence of re-prioritization.
This has been seen to help in most server style loads even on
small machines.  The idea is that the standard kernel has to
deal with screen re-draw at mouse interrupt speed. This will look
perceptibly jerky if context switches on a 1/10s second time
scale.  But these reates are fine for a database server, since
you'd be better off finishing work before kicking threads off of
CPUs. If you want to do more with the dispatch table.  Solaris
2.5.1. and 2.6 for Starfire load the same large quanta custom
dispatch tables in various ways.

OS upgrades

Always verify each entry, and try to drop if possible when you
go to a new version of the OS.  The same is true for kernel
patches.

maxcontig

maxcontig is an attribute of Solaris filesystems that defines
prefetch policy.. It is nice to prefetch for big sequential
access.  However, It can have a major negative impact on memory
if read ahead is unconstrained. When running a database on
filesystem, the more effective way to go is with maxcontig = 1,
totally turn off filesystem pre-fetch. The database and disk (to
track buffers) do a better job of pre-fetch. The hardware RAID
(A3500 and T3) devices do prefetch which is another reason to
disable or reduce it on the filesystem.

maxcontig is set via device-driver specific defaults:  sd =
maxphys, ssd = 1 MB, vxvm = stripe_width, ods = maxphys.

ndd Tuning

ndd tuning of the devices /dev/tcp /dev/ge fall in the category
of kernel tuning. tcp_xmit_hiwat is the most common subject of
tuning.  The default under most kernels is small, and therefore
you can find places that recommend running around 256k if you
have interfaces faster than 100BaseT.  The default was 256k in
Solaris 2.6.

Check out :
http://www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html#etcsystem
This is a nice page that may offer a different perspective or
reinforce some of the reccommendations found here.

