Tips&Tricks: LINUX SERVER HACKS

LINUX SERVER HACKS
Rob Flickenger
Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • TokyoLinux Server Hacks
by Rob Flickenger
Copyright © 2003 O’Reilly Media, Inc. All rights reserved.
Printed in the United States of America.

PublishedbyO’ReillyMedia,Inc.,1005GravensteinHighwayNorth,Sebastopol,CA95472.
O’ReillyMedia,Inc.booksmaybepurchasedforeducational,business,orsalespromotional
use. Online editions are also available for most titles (safari.oreilly.com). For more
information, contact our corporate/institutional sales department: (800) 998-9938 or
corporate@oreilly.com.
Editor: Dale Dougherty
Series Editor: Rael Dornfest
Executive Editor: Dale Dougherty
Production Editor: Sarah Sherman
Cover Designer: Edie Freedman
Interior Designer: David Futato
Printing History:
January 2003:First Edition.
Nutshell Handbook, the Nutshell Handboook logo, and the O’Reilly logo are registered
trademarksofO’ReillyMedia,Inc.Theassociationbetweentheimageofanaxandthetopic
of Linux servers is a trademark of O’Reilly Media, Inc.
Thetrademarks“HacksBooks”and“TheHacksSeries,”andrelatedtradedress,areowned
by O’Reilly Media, Inc., in the United States and other countries, and may not be used
without written permission.
Manyofthedesignationsusedbymanufacturersandsellerstodistinguishtheirproductsare
claimed as trademarks. Where those designations appear in this book, and O’Reilly Media,
Inc. was aware of a trademark claim, the designations have been printed in caps or initial
caps. All other trademarks are property of their respective owners.
While every precaution has been taken in the preparation of this book, the publisher and
authorassumenoresponsibilityforerrorsoromissions,orfordamagesresultingfromtheuse
of the information contained herein.
ISBN13: 978-0-596-00461-3
This book uses RepKover™ , a durable and flexible lay-flat binding.
[C] [4/08]
Contents
Credits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
How to Become a Hacker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Chapter1.Server Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.Removing Unnecessary Services 3
2.Forgoing the Console Login 6
3.Common Boot Parameters 7
4.Creating a Persistent Daemon with init 9
5.n>&m: Swap Standard Output and StandardError 10
6.Building Complex Command Lines 12
7.Working with Tricky Files in xargs 15
8.Immutable Files in ext2/ext3 17
9.Speeding Up Compiles 19
10.At Home in Your Shell Environment 20
11.Finding and Eliminating setuid/setgid Binaries 23
12.Make sudo Work Harder 25
13.Using a Makefile to Automate Admin Tasks 27
14.Brute Forcing Your New Domain Name 29
15.Playing Hunt the Disk Hog 30
16.Fun with /proc 31
17.Manipulating Processes Symbolically withprocps 34
18.Managing System Resources per Process 36
19.Cleaning Up after Ex-Users 38iv | Contents
20.Eliminating Unnecessary Drivers from the Kernel 40
21.Using Large Amounts of RAM 42
22.hdparm: Fine Tune IDE Drive Parameters 43
Chapter2.Revision Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
23.Getting Started with RCS 48
24.Checking Out a Previous Revision in RCS 49
25.Tracking Changes with rcs2log 50
26. Getting Started with CVS 52
27.CVS: Checking Out a Module 54
28.CVS: Updating Your Working Copy 55
29.CVS: Using Tags 56
30.CVS: Making Changes to a Module 57
31.CVS: Merging Files 58
32.CVS: Adding and Removing Files and Directories 58
33.CVS: Branching Development 59
34.CVS: Watching and Locking Files 60
35.CVS: Keeping CVS Secure 60
36.CVS: Anonymous Repositories 62
Chapter3.Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
37.Backing Up with tar over ssh 65
38.Using rsync over ssh 66
39.Archiving with Pax 67
40.Backing Up Your Boot Sector 72
41.Keeping Parts of Filesystems in sync with rsync 74
42.Automated Snapshot-Style Incremental Backups with rsync 79
43.Working with ISOs and CDR/CDRWs 84
44.Burning a CD Without Creating an ISO File 86
Chapter4.Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
45.Creating a Firewall from the Command Line of any Server 88
46.Simple IP Masquerading 91
47.iptables Tips & Tricks 92
48.Forwarding TCP Ports to Arbitrary Machines 94
49.Using Custom Chains in iptables 96Contents | v
50.Tunneling: IPIP Encapsulation 97
51.Tunneling: GRE Encapsulation 99
52.Using vtun over ssh to Circumvent NAT 101
53.Automatic vtund.conf Generator 106
Chapter5.Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
54.Steering syslog 111
55.Watching Jobs with watch 114
56.What’s Holding That Port Open? 115
57.Checking On Open Files and Sockets with lsof 116
58.Monitor System Resources with top 119
59.Constant Load Average Display in the Titlebar 120
60.Network Monitoring with ngrep 121
61.Scanning Your Own Machines with nmap 123
62.Disk Age Analysis 125
63.Cheap IP Takeover 127
64.Running ntop for Real-Time Network Stats 129
65.Monitoring Web Traffic in Real Time with httptop 132
Chapter6.SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
66.Quick Logins with ssh Client Keys 139
67.Turbo-mode ssh Logins 141
68.Using ssh-Agent Effectively 142
69.Running the ssh-Agent in a GUI 144
70.X over ssh 145
71.Forwarding Ports over ssh 146
Chapter7.Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
72.Get Settled in Quickly with movein.sh 149
73.Global Search and Replace with Perl 151
74.Mincing Your Data into Arbitrary Chunks (in bash) 153
75.Colorized Log Analysis in Your Terminal 155
Chapter8.Information Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
76.Running BIND in a chroot Jail 158
77.Views in BIND 9 160
78.Setting Up Caching DNS with Authority for Local Domains 165vi | Contents
79.Distributing Server Load with Round-Robin DNS 167
80.Running Your Own Top-Level Domain 168
81.Monitoring MySQL Health with mtop 169
82.Setting Up Replication in MySQL 172
83.Restoring a Single Table from a Large MySQL Dump 175
84.MySQL Server Tuning 175
85.Using proftpd with a mysql Authentication Source 178
86.Optimizing glibc, linuxthreads, and the Kernel
for a Super MySQL Server 180
87.Apache Toolbox 182
88.Display the Full Filename in Indexes 185
89.Quick Configuration Changes with IfDefine 186
90.Simplistic Ad Referral Tracking 188
91.Mimicking FTP Servers with Apache 191
92.Rotate and compress Apache Server Logs 193
93.Generating an SSL cert and Certificate Signing Request 194
94.Creating Your Own CA 196
95.Distributing Your CA to Client Browsers 199
96.Serving multiple sites with the same DocumentRoot 201
97.Delivering Content Based on the Query String
Using mod_rewrite 203
98.Using mod_proxy on Apache for Speed 204
99.Distributing Load with Apache RewriteMap 206
100.Ultrahosting: Mass Web Site Hosting
with Wildcards, Proxy, and Rewrite 208
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

0
Credits
About the Author
RobFlickengerauthoredthemajorityofhacksinthisbook.Robhasworked
withLinuxsinceSlackware3.5.Hewaspreviouslythesystemadministrator
oftheO’ReillyNetwork(anall-Linuxshop,naturally)andistheauthorof
Building Wireless Community Networks, also by O’Reilly.
Contributors
Thefollowingpeoplecontributedtheirhacks,writing,andinspirationto
this book:
•RaelDornfest(“ApacheToolbox” [Hack#75])isamavenattheO’Reilly&
Associatesfocusingontechnologiesjustbeyondthepale.Heassesses,
experiments,programs,andwritesfortheO’Reillynetworkand
O’Reilly publications.
•SchuylerErle(contributedcodeforhttptop,mysql-table-restore,bal-
ance-push,find-whois,andvtundgen)is,byday,amild-mannered
InternetsystemsdeveloperforO’Reilly&Associates.Bynight,hecru-
sadesforjusticeandfreedomasafreesoftwarehackerandcommunity
networking activist.
•KevinHemenway(“QuickConfigurationChangeswithIfDefine” [Hack
#75],“SimplisticAdReferralTracking” [Hack#75],“MimickingFTPServ-
erswithApache” [Hack#75]),betterknownasMorbusIff,isthecreatorof
disobey.com,whichbillsitselfas“contentforthediscontented.”Pub-
lisheranddeveloperofmorehomecookingthanyoucouldeverimag-
ine,he’dlovetogiveyouaFryPanofIntellectupsidethehead.Politely,
of course. And with love.viii | Credits
•SeannHerdejurgen(“iptablesTips&Tricks” [Hack#44],“DiskAgeAnal-
ysis” [Hack#53])hasbeenworkingwithUnixsince1987andnowarchi-
tectshighavailabilitysolutionsasaseniorsystemsengineerwithD-
TechcorporationinDallas,Texas.Heholdsamaster’sdegreeincom-
putersciencefromTexasA&MUniversity.Hemaybereachedat:
http://seann.herdejurgen.com/.
•DruLavigne(“ArchivingwithPax” [Hack#36])isaninstructorataprivate
technicalcollegeinKingston,ONwheresheteachesthefundamentals
ofTCP/IPnetworking,routing,andsecurity.Hercurrenthobbies
includematchingeveryTCPandUDPportnumbertoitsassociated
application(s) and reading her way through all of the RFCs.
•CricketLiu(“ViewsinBIND9” [Hack#75])matriculatedattheUniversity
ofCalifornia’sBerkeleycampus,thatgreatbastionoffreespeech,unen-
cumberedUnix,andcheappizza.HeworkedforayearasDirectorof
DNSProductManagementforVeriSignGlobalRegistryServices,andis
a co-author of DNS and BIND, also published by O’Reilly & Associates.
•MikeRubel(“AutomatedSnapshot-StyleIncrementalBackupswith
rsync” [Hack#36], http://www.mikerubel.org)studiedmechanicalengineer-
ingatRutgersUniversity(B.S.1998)andaeronauticsatCaltech(M.S.
1999),whereheisnowagraduatestudent.HehasenjoyedusingLinux
andGNUsoftwareforseveralyearsinthecourseofhisnumericalmeth-
ods research.
•JenniferVesperman(alloftheCVSpiecesexcept“CVS:Anonymous
Repositories” [Hack#22] wereadaptedfromheronlineCVSpiecesfor
O’ReillyNet)contributestoopensourceasauser,writer,andoccasional
programmer.Hercodingexperiencerangesfromthehardwareinterface
toanHDLCcardtothehumaninterfaceofJavaGUIs.Jennisthecur-
rent coordinator and co-sysadmin for Linuxchix.org.
Acknowledgments
Iwouldliketothankmyfamilyandfriendsfortheirsupportandencourage-
ment.Thanksespeciallytomydadforshowingme“propertroubleshooting
technique”atsuchanearlyageandinevitablysettingmeonthepathofthe
Hacker before I had even seen a computer.
Ofcourse,thisbookwouldbenothingwithouttheexcellentcontributions
ofallthephenomenallytalentedhackerscontainedherein.Butofcourse,
ourhacksarebuiltbyhackingtheshouldersofgiants(tohorriblymixa
metaphor),anditismysincerehopethatyouwillinturntakewhatyou
learnhereandgoonebetter,andmostimportantly,telleveryonejusthow
you did it.ix
FOREWORD
How to Become a Hacker
TheJargonFilecontainsabunchofdefinitionsoftheterm“hacker,”most
havingtodowithtechnicaladeptnessandadelightinsolvingproblemsand
overcominglimits.Ifyouwanttoknowhowto become ahacker,though,
only two are really relevant.
Thereisacommunity,asharedculture,ofexpertprogrammersandnet-
workingwizardsthattracesitshistorybackthroughdecadestothefirst
time-sharingminicomputersandtheearliestARPAnetexperiments.The
membersofthiscultureoriginatedtheterm“hacker.”Hackersbuiltthe
Internet.HackersmadetheUnixoperatingsystemwhatitistoday.Hackers
runUsenet.HackersmaketheWebwork.Ifyouarepartofthisculture,if
youhavecontributedtoitandotherpeopleinitknowwhoyouareandcall
you a hacker, you’re a hacker.
Thehackermind-setisnotconfinedtothissoftware-hackerculture.There
arepeoplewhoapplythehackerattitudetootherthings,likeelectronicsor
music—actually,youcanfinditatthehighestlevelsofanyscienceorart.
Softwarehackersrecognizethesekindredspiritselsewhereandmaycall
them“hackers”too—andsomeclaimthatthehackernatureisreallyinde-
pendentoftheparticularmediumthehackerworksin.Butintherestofthis
document,wewillfocusontheskillsandattitudesofsoftwarehackers,and
the traditions of the shared culture that originated the term “hacker.”
Thereisanothergroupofpeoplewholoudlycallthemselveshackers,but
aren’t.Thesearepeople(mainlyadolescentmales)whogetakickoutof
breakingintocomputersandbreakingthephonesystem.Realhackerscall
thesepeople“crackers”andwantnothingtodowiththem.Realhackers
mostlythinkcrackersarelazy,irresponsible,andnotverybright—being
abletobreaksecuritydoesn’tmakeyouahackeranymorethanbeingable
tohotwirecarsmakesyouanautomotiveengineer.Unfortunately,many
journalistsandwritershavebeenfooledintousingtheword“hacker”to
describe crackers; this irritates real hackers no end.x | How to Become a Hacker
The basic difference is this: hackers build things, crackers break them.
Ifyouwanttobeahacker,keepreading.Ifyouwanttobeacracker,goread
thealt.2600newsgroupandgetreadytodofivetotenintheslammerafter
findingoutyouaren’tassmartasyouthinkyouare.Andthat’sallI’mgoing
to say about crackers.
The Hacker Attitude
Hackerssolveproblemsandbuildthings,andtheybelieveinfreedomand
voluntarymutualhelp.Tobeacceptedasahacker,youhavetobehaveas
thoughyouhavethiskindofattitudeyourself.Andtobehaveasthoughyou
have the attitude, you have to really believe the attitude.
Butifyouthinkofcultivatinghackerattitudesasjustawaytogainaccep-
tanceintheculture,you’llmissthepoint.Becomingthekindofpersonwho
believesthesethingsisimportantfor you—forhelpingyoulearnandkeep-
ingyoumotivated.Aswithallcreativearts,themosteffectivewayto
becomeamasteristoimitatethemind-setofmasters—notjustintellectu-
ally but emotionally as well.
Or, as the following modern Zen poem has it:
To follow the path:
look to the master,
follow the master,
walk with the master,
see through the master,
become the master.
Soifyouwanttobeahacker,repeatthefollowingthingsuntilyoubelieve
them.
1. The world is full of fascinating problems waiting to be
solved.
Beingahackerisalotoffun,butit’sakindoffunthattakesalotofeffort.
Theefforttakesmotivation.Successfulathletesgettheirmotivationfroma
kindofphysicaldelightinmakingtheirbodiesperform,pushingthemselves
pasttheirownphysicallimits.Similarly,tobeahackeryouhavetogeta
basicthrillfromsolvingproblems,sharpeningyourskills,andexercising
your intelligence.
Ifyouaren’tthekindofpersonthatfeelsthiswaynaturally,you’llneedto
becomeoneinordertomakeitasahacker.Otherwiseyou’llfindyourhack-
ing energy is zapped by distractions like sex, money, and social approval.How to Become a Hacker | xi
(Youalsohavetodevelopakindoffaithinyourownlearningcapacity—a
beliefthateventhoughyoumaynotknowallofwhatyouneedtosolvea
problem,ifyoutacklejustapieceofitandlearnfromthat,you’lllearn
enough to solve the next piece—and so on, until you’re done.)
2. No problem should ever have to be solved twice.
Creativebrainsareavaluable,limitedresource.Theyshouldn’tbewasted
onre-inventingthewheelwhentherearesomanyfascinatingnewproblems
waiting out there.
Tobehavelikeahacker,youhavetobelievethatthethinkingtimeofother
hackersisprecious—somuchsothatit’salmostamoraldutyforyouto
shareinformation,solveproblems,andthengivethesolutionsawayjustso
otherhackerscansolve new problemsinsteadofhavingtoperpetuallyre-
address old ones.
(Youdon’thavetobelievethatyou’reobligatedtogive all yourcreative
productaway,thoughthehackersthatdoaretheonesthatgetmostrespect
fromotherhackers.It’sconsistentwithhackervaluestosellenoughofitto
keepyouinfoodandrentandcomputers.It’sfinetouseyourhackingskills
tosupportafamilyorevengetrich,aslongasyoudon’tforgetyourloyalty
to your art and your fellow hackers while doing it.)
3. Boredom and drudgery are evil.
Hackers(andcreativepeopleingeneral)shouldneverbeboredorhaveto
drudgeatstupidrepetitivework,becausewhenthishappensitmeansthey
aren’tdoingwhatonlytheycando—solvenewproblems.Thiswastefulness
hurtseverybody.Thereforeboredomanddrudgeryarenotjustunpleasant
but actually evil.
Tobehavelikeahacker,youhavetobelievethisenoughtowanttoauto-
mateawaytheboringbitsasmuchaspossible,notjustforyourselfbutfor
everybody else (especially other hackers).
(Thereisoneapparentexceptiontothis.Hackerswillsometimesdothings
thatmayseemrepetitiveorboringtoanobserverasamind-clearingexer-
cise,orinordertoacquireaskillorhavesomeparticularkindofexperience
youcan’thaveotherwise.Butthisisbychoice—nobodywhocanthink
should ever be forced into a situation that bores them.)
4. Freedom is good.
Hackersarenaturallyanti-authoritarian.Anyonewhocangiveyouorders
canstopyoufromsolvingwhateverproblemyou’rebeingfascinatedxii | How to Become a Hacker
by—and,giventhewayauthoritarianmindswork,willgenerallyfind
someappallinglystupidreasontodoso.Sotheauthoritarianattitudehas
to be fought wherever you find it, lest it smother you and other hackers.
(Thisisn’tthesameasfightingallauthority.Childrenneedtobeguidedand
criminalsrestrained.Ahackermayagreetoacceptsomekindsofauthority
inordertogetsomethinghewantsmorethanthetimehespendsfollowing
orders.Butthat’salimited,consciousbargain;thekindofpersonalsurren-
der authoritarians want is not on offer.)
Authoritariansthriveoncensorshipandsecrecy.Andtheydistrustvolun-
tarycooperationandinformation-sharing—theyonlylike“cooperation”
thattheycontrol.Sotobehavelikeahacker,youhavetodevelopaninstinc-
tivehostilitytocensorship,secrecy,andtheuseofforceordeceptionto
compel responsible adults. And you have to be willing to act on that belief.
5. Attitude is no substitute for competence.
Tobeahacker,youhavetodevelopsomeoftheseattitudes.Butcoppingan
attitudealonewon’tmakeyouahacker,anymorethanitwillmakeyoua
championathleteorarockstar.Becomingahackerwilltakeintelligence,
practice, dedication, and hard work.
Therefore,youhavetolearntodistrustattitudeandrespectcompetenceof
everykind.Hackerswon’tletposerswastetheirtime,buttheyworship
competence—especiallycompetenceathacking,butcompetenceatany-
thingisgood.Competenceatdemandingskillsthatfewcanmasterisespe-
ciallygood,andcompetenceatdemandingskillsthatinvolvemental
acuteness, craft, and concentration is best.
Ifyoureverecompetence,you’llenjoydevelopingitinyourself—thehard
workanddedicationwillbecomeakindofintenseplayratherthandrudg-
ery. That attitude is vital to becoming a hacker.
Thecompleteessaycanbefoundonlineat http://www.catb.org/~esr/faqs/
hacker-howto.html andinanappendixtothe“TheCathedralandthe
Bazaar" book (O’Reilly.)
—Eric S. Raymond
EricS.RaymondistheauthoroftheNewHacker’sDictionary,basedontheJargon
File,andthefamous“CathedralandtheBazaar”essaythatservedasacatalystfor
theOpenSourcemovement.ThetextinthisForewordisanexcerptfromhis1996
essay,“Whatisahacker?”Raymondarguesthathackersareingeniousatsolving
interesting problems, an idea that is the cornerstone of O’Reilly’s Hacks series.xiii
Preface
A hacker does for love what others
would not do for money.
—/usr/games/fortune
Theword hack hasmanyconnotations.A“goodhack”makesthebestofthe
situationofthemoment,usingwhateverresourcesareathand.An“ugly
hack”approachesthesituationinthemostobscureandleastunderstand-
ableway,althoughmany“goodhacks”mayalsoappearunintelligibletothe
uninitiated.
Theeffectivenessofahackisgenerallymeasuredbyitsabilitytosolveapar-
ticulartechnicalproblem,inverselyproportionaltotheamountofhuman
effortinvolvedingettingthehackrunning.Somehacksarescalableand
someareevensustainable.Thelongestrunningandmostgenerallyaccepted
hacksbecomestandardsandcausemanymorehackstobeinvented.Agood
hack lasts until a better hack comes along.
Ahackrevealstheinterfacebetweentheabstractandwonderfullycomplex
mindofthedesigner,andtheindisputableandvulgarexperienceofhuman
needs.Sometimes,hacksmaybeuglyandonlyexistbecausesomeonehadan
itchthatneededscratching.Totheengineer,ahackistheultimateexpression
oftheDo-It-Yourselfsentiment:nooneunderstandshowahackcametobe
betterthanthepersonwhofeltcompelledtosolvetheprobleminthefirst
place.Ifapersonwithabentforproblemsolvingthinksagivenhackisugly,
thentheyarealmostalwaysirresistiblymotivatedtogoonebetter—andhack
the hack, something that we encourage the readers of this book to do.
Intheend,eventhemostcapableserver,withthemostRAMandrunning
thefastest(andmostfree)operatingsystemontheplanet,isstilljustafancy
back-scratcherfixingtheitchofthemoment,untilabetter,fasterand
cheaper back-scratcher is required.
Download from Wow! eBook <www.wowebook.com>xiv | Preface
Wheredoesallofthispseudo-philosophicalramblinggetyou?Hopefully,
thisbackgroundwillgiveyousomeideaofthemindsetthatpromptedthe
compilingofthiscollectionofsolutionsthatwecallLinuxServerHacks.
Someareshortandsimple,whilesomearequitecomplex.Allofthesehacks
aredesignedtosolveaparticulartechnicalproblemthatthedesignersimply
couldn’tletgowithout“scratching.”Ihopethatsomeofthemwillbe
directlyapplicabletoan“itch”ortwothatyoumayhavefeltyourselfasa
new or experienced administrator of Linux servers.
How This Book Is Organized
Acompetentsysadminmustbeajack-of-all-trades.Tobetrulyeffective,
you’llneedtobeabletohandleeveryproblemthesystemthrowsatyou,
frompowerontohalt.Toassistyouinthetimeinbetween,Ipresentthis
collectionoftime-savingandnovelapproachestodailyadministrativetasks.
• ServerBasics beginsbylookingatsomeofthemostcommonsortsof
tasksthatadminsencounter:manipulatingthebootprocess,effectively
workingwiththecommandline,automatingcommontasks,watching
(andregulating)howsystemresourcesareused,andtuningvarious
piecesoftheLinuxkerneltomakeeverythingrunmoreefficiently.This
isn’tanintroductiontosystemadministrationbutalookatsomevery
effectiveandnon-obvioustechniquesthatevenseasonedsysadminsmay
have overlooked.
• RevisionControl givesacrash-courseinusingtwofundamentalrevision
controlsystems,RCSandCVS.Beingabletorecallarbitraryprevious
revisionsofconfigurationfiles,sourcecode,anddocumentationisa
criticalabilitythatcansaveyourjob.Toomanyprofessionaladminsare
lackinginrevisioncontrolbasics(preferringinsteadtomaketheinevita-
ble,butunsupportable .old or .orig backup).Thissectionwillgetyou
upandrunningquickly,givingyoucommandsandinstructionsthatare
succinct and to the point.
•Thenextsection, Backups,looksatquickandeasymethodsforkeeping
sparecopiesofyourdata.Ipayparticularattentiontonetworkback-
ups, rsync,andworkingwithISOs.I’lldemonstratesomeoftheenor-
mousflexibilityofstandardsystembackuptoolsandevenpresentone
wayofimplementingregular“snapshot”revisionsofafilesystem(with-
out requiring huge amounts of storage).
• Networking ismyfavoritesectionofthisentirebook.Thefocusisn’ton
basicfunctionalityandrouting,butinsteadlooksatsomeobscurebut
insanelyusefultechniquesformakingnetworksbehaveinunexpected
ways.I’llsetupvariouskindsofIPtunnels(bothencryptedandPreface | xv
otherwise),workwithNAT,andshowsomeadvancedfeaturesthat
allowforinterestingbehaviorbasedonallkindsofparameters.Didyou
everwanttodecidewhattodowithapacketbasedonitsdatacon-
tents? Take a look at this section.
• Monitoring isaneclecticmixoftipsandtoolsforfindingoutexactly
whatyourserverisupto.Itlooksatsomestandard(andsomeabso-
lutelyrequired“optional”)packagesthatwilltellyouvolumesabout
whoisusingwhat,when,andhowonyournetwork.Italsolooksata
coupleofwaystomitigateinevitableservicefailuresandevenhelp
detectwhennaughtypeopleattempttodonot-so-nicethingstoyour
network.
•Trulyafontofhackeryuntoitself,the SSH sectiondescribesallsortsof
niftyusesfor ssh,thecryptographicallystrong(andwonderfullyflexi-
ble)networkingtool.Thereareacoupleofversionsof ssh availablefor
Linux,andwhilemanyoftheexampleswillworkinallversions,they
are all tested and known to work with OpenSSH v3.4p1.
• Scripting providesashortdigressionbylookingatacoupleofoddsand
endsthatsimplycouldn’tfitonasinglecommandline.Thesehackswill
saveyoutimeandwillhopefullyserveasexamplesofhowtodosome
nifty things in shell and Perl.
• InformationServices presentsthreemajorapplicationsforLinux: BIND9,
MySQL,and Apache.Thissectionassumesthatyou’rewellbeyondbasic
installationofthesepackages,andarelookingforwaystomakethem
delivertheirservicesfasterandmoreefficiently,withouthavingtodoalot
ofworkyourself.Youwillseemethodsforgettingyourserverrunning
quickly,helpingitscaletoverylargeinstallationsandbehaveinallsorts
of slick ways that save a lot of configuration and maintenance time.
How to Use This Book
Youmayfinditusefultoreadthisbookfromcovertocover,asthehacksdo
buildoneachotherabitfrombeginningtoend.However,eachhackis
designedtostandonitsownasaparticularexampleofonewaytoaccom-
plishaparticulartask.Tothatend,Ihavegroupedtogetherhacksthatfita
particularthemeintosections,butIdocross-referencequiteabitbetween
hacksfromdifferentsections(andalsotomoredefinitiveresourcesonthe
subject).Don’tconsideragivensectionasacut-and-driedchapterwithrig-
idlydefinedsubjectboundariesbutmoreasaconvenientwayofcollecting
similar(andyetindependent)hacks.Youmaywanttoreadthisbookmuch
likethewaymostpeoplebrowsewebpagesonline:followwhateverinter-
estsyou,andifyougetlost,followthelinkswithinthepiecetofindmore
information.xvi | Preface
Conventions Used in This Book
The following is a list of the typographical conventions used in this book:
Italic
Usedtoindicatenewterms,URLs,filenames,fileextensions,directo-
ries, commands and options, and program names.
Constant Width
Usedtoshowcodeexamples,thecontentsoffiles,ortheoutputfrom
commands.
Constant Width Bold
Usedinexamplesandtablestoshowcommandsorothertextthat
should be typed literally.
Constant Width Italic
Usedinexamplesandtablestoshowtextthatshouldbereplacedwith
user-supplied values.
Thethermometericons,foundnexttoeachhack,indicatetherelativecom-
plexity of the hack:
How to Contact Us
Wehavetestedandverifiedtheinformationinthisbooktothebestofour
ability,butyoumayfindthatfeatureshavechanged(oreventhatwehave
mademistakes!).Pleaseletusknowaboutanyerrors,inaccuracies,bugs,mis-
leading or confusing statements, and typos that you find in this book.
You can write to us at:
O’Reilly & Associates, Inc.
1005 Gravenstein Hwy N.
Sebastopol, CA 95472
(800) 998-9938 (in the U.S. or Canada)
(707) 829-0515 (international/local)
(707) 829-0104 (fax)
To ask technical questions or to comment on the book, send email to:
bookquestions@oreilly.com
Visitthewebpagefor LinuxServerHacks tofindadditionalsupportinfor-
mation, including examples and errata. You can find this page at:
http://www.oreilly.com/catalog/linuxsvrhack
beginner moderate expertPreface | xvii
For more information about this book and others, see the O’Reilly web site:
http://www.oreilly.com
Gotta Hack?
Gotagoodhackyou’dliketosharewithothers?GototheO’ReillyHacks
web site at:
http://hacks.oreilly.com
You’llfindbook-relatedresources,samplehacksandnewhackscontrib-
utedbyusers.You’llfindinformationaboutadditionalbooksintheHacks
series.Download from Wow! eBook <www.wowebook.com>1
Chapter1 CHAPTER ONE
Server Basics
Hacks #1–22
ArunningLinuxsystemisacomplexinteractionofhardwareandsoftware
whereinvisibledaemonsdotheuser’sbidding,carryingoutarcanetasksto
thebeatofthedrumoftheuncompromisingtaskmastercalledtheLinux
kernel.
ALinuxsystemcanbeconfiguredtoperformmanydifferentkindsoftasks.
Whenrunningasadesktopmachine,thevisibleportionofLinuxspends
muchofitstimecontrollingagraphicaldisplay,paintingwindowsonthe
screen,andrespondingtotheuser’severygestureandcommand.Itmust
generallybeaveryflexible(andentertaining)system,wheregoodrespon-
siveness and interactivity are the critical goals.
Ontheotherhand,aLinuxservergenerallyisdesignedtoperformacouple
oftasks,nearlyalwaysinvolvingthesqueezingofinformationdownanet-
workconnectionasquicklyaspossible.WhileprettyscreensaversandGUI
featuresmaybecriticaltoasuccessfuldesktopsystem,thesuccessfulLinux
serverisahighperformanceappliancethatprovidesaccesstoinformationas
quicklyandefficientlyaspossible.Itpullsthatinformationfromsomesort
ofstorage(likethefilesystem,adatabase,orsomewhereelseonthenet-
work)anddeliversthatinformationoverthenetworktowhomever
requestedit,beitahumanbeingconnectedtoawebserver,ausersittingin
a shell, or over a port to another server entirely.
Itisunderthesecircumstancesthatasystemadministratorfindstheir
responsibilitieslyingsomewherebetweendeityandjanitor.Ultimately,the
sysadmin’sjobistoprovideaccesstosystemresourcesasquickly(andequi-
tably)aspossible.Thisjobinvolvesboththeabilitytodesignnewsystems
(thatmayormaynotberootedinsolutionsthatalreadyexist)andthetal-
ent(andthestomach)forcleaningupafterpeoplewhousethatsystem
without any concept of what “resource management” really means.2 | Server Basics
Themostsuccessfulsysadminsremovethemselvesfromthepathofaccess
tosystemresourcesandletthemachinesdoallofthework.Asauser,you
knowthatyoursysadminiseffectivewhenyouhavethetoolsthatyouneed
togetthejobdoneandyouneverneedtoaskyoursysadminforanything.
Topulloff(thatis,tohack)thisimpossiblesoundingtaskrequiresthatthe
sysadminanticipatewhattheusers’needswillbeandmakeefficientuseof
the resources that are available.
Tobeginwith,I’llpresentwaystooptimizeLinuxtoperformonlythework
thatisrequiredtogetthejobdoneandnotwastecyclesdoingworkthat
you’renotinterestedindoing.You’llseesomeexamplesofhowtogetthe
systemtodomoreoftheworkofmaintainingitselfandhowtomakeuseof
someofthemoreobscurefeaturesofthesystemtomakeyourjobeasier.
Partsofthissection(particularlyCommandLineandResourceManage-
ment)includetechniquesthatyoumayfindyourselfusingeverydaytohelp
buildapictureofhowpeopleareusingyoursystemandwaysthatyou
might improve it.
ThesehacksassumethatyouarealreadyfamiliarwithLinux.Inparticular,
youshouldalreadyhaverootonarunningLinuxsystemavailablewith
whichtoexperimentandshouldbecomfortablewithworkingonthesys-
temfromthecommandline.Youshouldalsohaveagoodworkingknowl-
edgeofnetworksandstandardnetworkservices.WhileIhopethatyouwill
findthesehacksinformative,theyarecertainlynotagoodintroductionto
Linuxsystemadministration.Forin-depthdiscussionongoodadministra-
tivetechniques,Ihighlyrecommendthe LinuxNetworkAdministrator’s
Guide and Essential System Administration, both by O’Reilly and Associates.
Thehacksinthischapteraregroupedtogetherintothefollowingfivecate-
gories:BootTime,CommandLine,Automation,ResourceManagement,
and Kernel Tuning.
Boot Time
1. Finetuneyourservertoprovideonlytheservicesyoureallywantto
serve
2. Forgoing the Console Login
3. Common Boot Parameters
4. Creating a Persistent Daemon with init
Command Line
5. n>&m: Swap Standard Output and StandardError
6. Building Complex Command Lines
7. Working with Tricky Files in xargsRemoving Unnecessary Services#1
Server Basics | 3
HACK
8. Immutable Files in ext2/ext3
9. Speeding Up Compiles
Automation
10. At home in your shell environments
11. Finding and eliminating setuid/setgid binaries
12. Make sudo work harder for you
13. Using a Makefile to automate admin tasks
14. Brute forcing your new domain name
Resource Management
15. Playing Hunt the Disk Hog
16. Fun with /proc
17. Manipulating processes symbolically with procps
18. Managing system resources per process
19. Cleaning up after ex-users
Kernel Tuning
20. Eliminating unnecessary drivers from the kernel
21. Using large amounts of RAM
22. hdparm: fine tune IDE drive parameters
HACK
#1
Removing Unnecessary Services Hack #1
Fine tune your server to provide only the services you really want to serve
Whenyoubuildaserver,youarecreatingasystemthatshouldperformits
intendedfunctionasquicklyandefficientlyaspossible.Justasapaintmixer
hasnorealbusinessbeingincludedasanespressomachineattachment,
extraneousservicescantakeupresourcesand,insomecases,causeareal
messthatiscompletelyunrelatedtowhatyouwantedtheservertodointhe
firstplace.ThisisnottosaythatLinuxisincapableofservingasbothatop-
notchpaintmixerandmakingagoodcupofcoffeesimultaneously—justbe
surethatthisisexactlywhatyouintendbeforeturningyourserverlooseon
the world (or rather, turning the world loose on your server).
Whenbuildingaserver,youshouldcontinuallyaskyourself:whatdoI
reallyneedthismachinetodo?DoIreallyneedFTPservicesonmyweb
server?ShouldNFSberunningonmyDNSserver,evenifnosharesare
exported?DoIneedtheautomountertorunifImountallofmyvolumes
statically?4 | Server Basics
#1 Removing Unnecessary Services
HACK
Togetanideaofwhatyourserverisupto,simplyruna psax.Ifnobodyis
loggedin,thiswillgenerallytellyouwhatyourserveriscurrentlyrunning.
Youshouldalsoseewhatprogramsforwhichyour inetd isacceptingcon-
nections,witheithera grep-v^#/etc/inetd.conf or(moretothepoint)
netstat-lp.Thefirstcommandwillshowalluncommentedlinesinyour
inetd.conf,whilethesecond(whenrunasroot)willshowallofthesockets
thatareintheLISTENstate,andtheprogramsthatarelisteningoneach
port.Ideally,youshouldbeabletoreducetheoutputofa psax toapageof
information or less (barring preforking servers like httpd, of course).
Herearesomenotorious(andtypicallyunnecessary)servicesthatare
enabled by default in many distributions:
portmap, rpc.mountd, rpc.nfsd
TheseareallpartoftheNFSsubsystem.AreyourunninganNFS
server?DoyouneedtomountremoteNFSshares?Unlessyouanswered
yes toeitherofthesequestions,youdon’tneedthesedaemonsrunning.
Reclaimtheresourcesthatthey’retakingupandeliminatethepotential
security risk.
smbd and nmbd
ThesearetheSambadaemons.DoyouneedtoexportSMBsharesto
Windowsboxes(orothermachines)?Ifnot,thentheseprocessescanbe
safely killed.
automount
Theautomountercanbehandytobringupnetwork(orlocal)filesys-
temsondemand,eliminatingtheneedforrootprivilegeswhenaccess-
ingthem.Thisisespeciallyhandyonclientdesktopmachines,wherea
userneedstouseremovablemedia(suchasCDsorfloppies)orto
accessnetworkresources.Butonadedicatedserver,theautomounteris
probablyunnecessary.Unlessyourmachineisprovidingconsoleaccess
orremotenetworkshares,youcan kill theautomounter(andsetupall
of your mounts statically, in /etc/fstab).
named
Areyourunninganameserver?Youdon’tneed named runningifyou
onlyneedtoresolvenetworknames;that’swhat /etc/resolv.conf andthe
bindlibrariesarefor.Unlessyou’rerunningnameservicesforother
machines,orarerunningacachingDNSserver(see“SettingUpCaching
DNSwithAuthorityforLocalDomains” [Hack#78]),then named isn’t
needed.Removing Unnecessary Services#1
Server Basics | 5
HACK
lpd
Doyoueverprinttothismachine?Chancesare,ifit’sservingInternet
resources,itshouldn’tbeacceptingprintrequestsanyway.Removethe
printer daemon if you aren’t planning on using it.
inetd
Doyoureallyneedtorunanyservicesfrom inetd?Ifyouhave ssh run-
ninginstandalonemode,andareonlyrunningstandalonedaemons
(suchasApache,BIND,MySQL,orProFTPD)then inetd maybe
superfluous.Intheveryleast,reviewwhichservicesarebeingaccepted
withthe grep command grep-v^#/etc/inetd.conf.Ifyoufindthat
everyservicecanbesafelycommentedout,thenwhyrunthedaemon?
Removeitfromthebootprocess(eitherbyremovingitfromthesys-
tem rc’s or with a simple chmod -x /usr/sbin/inetd).
telnet, rlogin, rexec, ftp
Theremotelogin,execution,andfiletransferfunctionalityofthesevener-
abledaemonshaslargelybeensupplantedby ssh and scp,theircrypto-
graphicallysecureandtremendouslyflexiblecounterparts.Unlessyou
havea really goodreasontokeepthemaround,it’sagoodideatoelimi-
natesupportfortheseonyoursystem.Ifyoureallyneedtosupport ftp
connections,youmighttrythe mod_sql pluginfor proftpd (see“Using
proftpd with a mysql Authentication Source” [Hack #85]).
finger, comsat, chargen, echo, identd
The finger and comsat servicesmadesenseinthedaysofanopenInter-
net,whereuserswerecuriousbutgenerallywell-intentioned.Inthese
daysofstealthportscansandremotebufferoverflowexploits,running
extraneousservicesthatgiveawayinformationaboutyourserverisgen-
erallyconsidereda bad idea.The chargen and echo portswereonce
goodfortestingnetworkconnectivity,butarenowtooinvitingfora
randommiscreanttofiddlewith(andperhapsconnecttoeachotherto
drive up server load quickly and inexpensively).
Finally,the identd servicewasonceameaningfulandimportantsourceof
information,providingremoteserverswithanideaofwhichuserswere
connectingtotheirmachines.Unfortunately,inthesedaysoflocalroot
exploitsanddesktopLinuxmachines,installingan identd that(perishthe
thought!)actually lies aboutwhoisconnectedhasbecomesocommon
thatmostsitesignoretheauthorinformationanyway.Since identd isa
notoriously shaky source of information, why leave it enabled at all?
Toeliminateunnecessaryservices,firstshutthemdown(eitherbyrunning
servicestop in /etc/rc.d/init.d/,removingthemfrom /etc/inetd.conf,orby
killingthemmanually).Thentobesurethattheydon’tstartagainthenext6 | Server Basics
#2 Forgoing the Console Login
HACK
timethemachinereboots,removetheirentryfrom /etc/rc.d/*.Onceyouhave
yoursystemtrimmeddowntoonlytheservicesyouintendtoserve,reboot
the machine and check the process table again.
Ifyouabsolutelyneedtoruninsecureservicesonyourmachine,thenyou
shouldusetcpwrappersorlocalfirewallingtolimitaccesstoonlythe
machines that absolutely need it.
See also:
•“Creating a Firewall from the Command Line of any Server” [Hack #45]
•“Setting Up Caching DNS with Authority for Local Domains” [Hack #78]
•“Using proftpd with a mysql Authentication Source” [Hack #85]
HACK
#2
Forgoing the Console Login Hack #2
All of the access, none of the passwords
Itwillhappentoyouoneday.You’llneedtoworkonamachineforafriend
orclientwhohas“misplaced”therootpasswordonwhichyoudon’thave
an account.
Ifyouhaveconsoleaccessanddon’tmindrebooting,traditionalwisdom
beckonsyoutobootupinsingleusermode.Naturally,afterhittingControl-
Alt-Delete,yousimplywaitforittoPOSTandthenpasstheparameter
single to the booting kernel. For example, from the LILO prompt:
LILO: linux single
Onmanysystems,thiswillhappilypresentyouwitharootshell.Buton
somesystems(notablyRedHat),you’llrunintothedreadedemergency
prompt:
Give root password for maintenance
(or type Control-D for normal startup)
Ifyouknewtherootpassword,youwouldn’tbehere!Ifyou’relucky,the
init scriptwillactuallyletyouhit^Catthisstageandwilldropyoutoa
rootprompt.Butmost init processesare“smarter”thanthat,andtrap^C.
Whattodo?Ofcourse,youcouldalwaysbootfromarescuediskandreset
thepassword,butsupposeyoudon’thaveonehandy(orthatthemachine
doesn’t have a CD-ROM drive).
Allisnotlost!Ratherthanriskrunningintotheabovemess,let’smodifythe
systemwithextremeprejudice,rightfromthestart.Again,fromtheLILO
prompt:
LILO: linux init=/bin/bashCommon Boot Parameters#3
Server Basics | 7
HACK
Whatdoesthisdo?Ratherthanstart /sbin/init andproceedwiththeusual /etc/
rc.d/* procedure,we’retellingthekerneltosimplygiveusashell.Nopass-
words,nofilesystemchecks(andforthatmatter,notmuchofastartingenvi-
ronment!) but a very quick, shiny new root prompt.
Unfortunately,that’snotquiteenoughtobeabletorepairyoursystem.The
rootfilesystemwillbemountedread-only(sinceitnevergotachancetobe
checkedandremountedread/write).Also,networkingwillbedown,and
noneoftheusualsystemdaemonswillberunning.Youdon’twanttodo
anythingmorecomplicatedthanresettingapassword(ortweakingafileor
two)atapromptlikethis.Aboveall:don’thit^DortypeExit!Yourlittle
shell(plusthekernel)constitutestheentirerunningLinuxsystematthe
moment.So,howcanyoumanipulatethefilesysteminthissituation,ifitis
mounted read-only? Try this:
# mount -o remount,rw /
Thatwillforcetherootfilesystemtoberemountedread-write.Youcannow
type passwd tochangetherootpassword(andiftheoriginaladminlostthe
password,considertheramificationsofgivingthemaccesstothenewone.If
youweretheoriginaladmin,considerwritingitininvisibleinkonapost-it
noteandstickingittoyourscreen,orstitchingitintoyourunderwear,or
maybe even taking up another hobby).
Oncethepasswordisreset,DONOTREBOOT.Sincethereisno init run-
ning,thereisnoprocessinplaceforsafelytakingthesystemdown.The
quickest way to shutdown safely is to remount root again:
# mount -o remount,ro /
Withtherootpartitionreadonly,youcanconfidentlyhittheResetbutton,
bring it up in single-user mode, and begin your actual work.
HACK
#3
Common Boot Parameters Hack #3
Manipulate kernel parameters at boot time
Aswesawin“ForgoingtheConsoleLogin” [Hack#2],itispossibletopass
parameterstothekernelattheLILOpromptallowingyoutochangethe
programthatisfirstcalledwhenthesystemboots.Changing init (withthe
init=/bin/bash line)isjustoneofmanyusefuloptionsthatcanbesetat
boot time. Here are more common boot parameters:
single
Boots up in single user mode.
root=
Changes the device that is mounted as /. For example:
root=/dev/sdc48 | Server Basics
#3 Common Boot Parameters
HACK
willbootfromthefourthpartitiononthethirdscsidisk(insteadof
whatever your boot loader has defined as the default).
hdX=
AdjustsIDEdrivegeometry.ThisisusefulifyourBIOSreportsincor-
rect information:
hda=3649,255,63 hdd=cdrom
Thisdefinesthemaster/primaryIDEdriveasa30GBharddriveinLBA
mode, and the slave/secondary IDE drive as a CD-ROM.
console=
Definesaserialportconsoleonkernelswithserialconsolesupport.For
example:
console=ttyS0,19200n81
Herewe’redirectingthekerneltologbootmessagestottyS0(thefirst
serialport),at19200baud,noparity,8databits,1stopbit.Notethat
togetanactualserialconsole(thatyoucanloginon),you’llneedto
add a line to /etc/inittab that looks something like this:
s1:12345:respawn:/sbin/agetty 19200 ttyS0 vt100
nosmp
DisablesSMPonakernelsoenabled.Thiscanhelpifyoususpectker-
nel trouble on a multiprocessor system.
mem=
Definesthetotalamountofavailablesystemmemory.See“UsingLarge
Amounts of RAM” [Hack #21].
ro
Mountsthe/partitionread-only(thisistypicallythedefault,andis
remounted read-write after fsck runs).
rw
Mountsthe/partitionread-write.Thisisgenerallyabadidea,unless
you’realsorunningthe init hack.Passyour init linealongwith rw,like
this:
init=/bin/bash rw
toeliminatetheneedforallofthatsilly mount-oremount,rw/ stuffin
“ForgoingtheConsoleLogin” [Hack#2].Congratulations,nowyou’ve
hacked a hack.
YoucanalsopassparametersforSCSIcontrollers,IDEdevices,soundcards,
andjustaboutanyotherdevicedriver.Everydriverisdifferent,andtypi-
callyallowsforsettingIRQs,baseaddresses,parity,speeds,optionsfor
auto-probing,andmore.Consultyouronlinedocumentationfortheexcru-
ciating details.Creating a Persistent Daemon with init#4
Server Basics | 9
HACK
See also:
•man bootparam
• /usr/src/linux/Documentation/*
HACK
#4
Creating a Persistent Daemon with init Hack #4
Make sure that your process stays up, no matter what
Thereareanumberofscriptsthatwillautomaticallyrestartaprocessifit
exits unexpectedly. Perhaps the simplest is something like:
$ while : ; do echo "Run some code here..."; sleep 1; done
Ifyourunaforegroundprocessinplaceofthat echo line,thentheprocessis
alwaysguaranteedtoberunning(or,atleast,itwill try torun).The : sim-
plymakesthe while alwaysexecute(andismoreefficientthanrunning /bin/
true,asitdoesn’thavetospawnanexternalcommandoneachiteration).
Definitelydo not runabackgroundprocessinplaceofthe echo,unlessyou
enjoyfillingupyourprocesstable(asthe while willthenspawnyourcom-
mandasmanytimesasitcan,oneeverysecond).Butasfarascoolhacksgo,
the while approach is fairly lacking in functionality.
Whathappensifyourcommandrunsintoanabnormalcondition?Ifitexits
immediately,thenitwillretryeverysecond,withoutgivinganyindication
thatthereisaproblem(unlesstheprocesshasitsownloggingsystemor
usessyslog).Itmightmakesensetohavesomethingwatchtheprocess,and
stop trying to respawn it if it returns too quickly after a few tries.
ThereisautilityalreadypresentoneveryLinuxsystemthatwilldothis
automaticallyforyou: init.Thesameprogramthatbringsupthesystemand
setsupyourterminalsisperfectlysuitedformakingsurethatprogramsare
always running. In fact, that is its primary job.
Youcanaddarbitrarylinesto /etc/inittab specifyingprogramsyou’dlike init
to watch for you:
zz:12345:respawn:/usr/local/sbin/my_daemon
The inittab lineconsistsofanarbitrary(butunique)twocharacteridentifica-
tionstring(inthiscase, zz),followedbytherunlevelsthatthisprogram
shouldberunin,thenthe respawn keyword,andfinallythefullpathtothe
command.Intheaboveexample,aslongas my_daemon isconfiguredtorun
intheforeground, init willrespawnanothercopywheneveritexits.After
makingchangesto inittab,besuretosendaHUPto init soitwillreloadits
configuration. One quick way to do this is:
# kill -HUP 1
Download from Wow! eBook <www.wowebook.com>10 | Server Basics
#5 n>&m: Swap Standard Output and StandardError
HACK
Ifthecommandrespawnstooquickly,then init willpostponeexecutionfor
a while, to keep it from tying up too many resources. For example:
zz:12345:respawn:/bin/touch /tmp/timestamp
Thiswillcausethefile /tmp/timestamp tobe touchedseveraltimesasec-
ond,until init decidesthatenoughisenough.Youshouldseethismessage
in /var/log/messages almost immediately:
Sep 8 11:28:23 catlin init: Id "zz" respawning too fast: disabled for 5
minutes
Infiveminutes, init willtrytorunthecommandagain,andifitisstill
respawning too quickly, it will disable it again.
Obviously,thismethodisfineforcommandsthatneedtorunasroot,but
whatifyouwantyourauto-respawningprocesstorunassomeotheruser?
That’s no problem: use sudo:
zz:12345:respawn:/usr/bin/sudo -u rob /bin/touch /tmp/timestamp
Nowthat touch willrunasrob,notasroot.Ifyou’retryingthesecom-
mandsasyoureadthis,besuretoremovetheexisting /tmp/timestamp
beforetryingthis sudo line.AftersendingaHUPto init,takealookatthe
timestamp file:
rob@catlin:~# ls -al /tmp/timestamp
-rw-r--r-- 1 rob users 0 Sep 8 11:28 /tmp/timestamp
Thetwodrawbackstousing init torunarbitrarydaemonsarethatyouneed
tocommentoutthelinein inittab ifyouneedtobringthedaemondown
(sinceitwilljustrespawnifyou kill it)andthatonlyrootcanaddentriesto
inittab.Butforkeepingaprocessrunningthatsimplymuststayupnomat-
ter what, init does a great job.
See also:
•“Make sudo Work Harder” [Hack #12]
HACK
#5
n>&m: Swap Standard Output and
StandardError Hack #5
Direct standard out and standard error to wherever you need them to go
Bydefault,acommand’sstandarderrorgoestoyourterminal.Thestandard
outputgoestotheterminalorisredirectedsomewhere(toafile,downa
pipe, into backquotes).
Sometimesyouwanttheopposite.Forinstance,youmayneedtosendacom-
mand’sstandardoutputtothescreenandgrabtheerrormessages(standard
error)withbackquotes.Or,youmightwanttosendacommand’sstandardn>&m: Swap Standard Output and StandardError#5
Server Basics | 11
HACK
outputtoafileandthestandarderrordownapipetoanerror-processing
command.Here’showtodothatintheBourneshell.(TheCshellcan’tdo
this.)
Filedescriptors0,1,and2arethestandardinput,standardoutput,and
standarderror,respectively.Withoutredirection,they’reallassociatedwith
theterminalfile /dev/tty.It’seasytoredirectanydescriptortoanyfile—if
youknowthefilename.Forinstance,toredirectfiledescriptorto errfile,
type:
$ command 2> errfile
You know that a pipe and backquotes also redirect the standard output:
$ command | ...\
$ var=`command`
Butthere’snofilenameassociatedwiththepipeorbackquotes,soyoucan’t
usethe2>redirection.Youneedtorearrangethefiledescriptorswithout
knowing the file (or whatever) that they’re associated with. Here’s how.
Let’sstartslowly.Bysendingbothstandardoutputandstandarderrorto
thepipeorbackquotes,theBourneshelloperator n>&m rearrangesthefiles
andfiledescriptors.Itsays“makefiledescriptor n pointtothesamefileas
filedescriptor m.”Let’susethatoperatoronthepreviousexample.We’ll
send standard error to the same place standard output is going:
$ command 2>&1 | ...
$ var=`command 2>&1`
Inboththoseexamples,2>&1means“sendstandarderror(filedescriptor2)
to the same place standard output (file descriptor 1) is going.” Simple, eh?
Youcanusemorethanoneofthose n>&m operators.Theshellreadsthem
left-to-right before it executes the command.
“Oh!”youmightsay,“Toswapstandardoutputandstandarderror—make
stderr go down a pipe and stdout go to the screen—I could do this!”
$ command 2>&1 1>&2 | ... (wrong...)
Sorry,Charlie.Whentheshellsees2>&11>&2,theshellfirstdoes2>&1.
You’veseenthatbefore—itmakesfiledescriptor2(stderr)gothesameplace
asfiledescriptor1(stdout).Then,theshelldoes1>&2.Itmakes stdout (1)
gothesameplaceas stderr (2),but stderr isalreadygoingthesameplaceas
stdout, down the pipe.
Thisisoneplacethattheotherfiledescriptors,3through9,comeinhandy.
Theynormallyaren’tused.Youcanuseoneofthemasa“holdingplace”to
rememberwhereanotherfiledescriptor“pointed.”Forexample,onewayto
readtheoperator3>&2is“make3pointthesameplaceas2”.Afteryouuse12 | Server Basics
#6 Building Complex Command Lines
HACK
3>&2tograbthelocationof2,youcanmake2pointsomewhereelse.
Then, make 1 point to where 2 used to (where 3 points now).
The command line you want is one of these:
$ command 3>&2 2>&1 1>&3 | ...
$ var=`command 3>&2 2>&1 1>&3`
Openfilesareautomaticallyclosedwhenaprocessexits.Butit’ssaferto
closethefilesyourselfassoonasyou’redonewiththem.Thatway,ifyou
forgetandusethesamedescriptorlaterforsomethingelse(forinstance,use
F.D.3toredirectsomeothercommand,orasubprocessusesF.D.3),you
won’trunintoconflicts.Use m<&- tocloseinputfiledescriptor m and m>
&- tocloseoutputfiledescriptor m.Ifyouneedtoclosestandardinput,use
<&- ; >&- will close standard output.
HACK
#6
Building Complex Command Lines Hack #6
Build simple commands into full-fledged paragraphs
for complex (but meaningful) reports
StudyingLinux(orindeedanyUnix)ismuchlikestudyingaforeignlan-
guage.Atsomemagicalpointinthecourseofone’sstudies,haltingmonosyl-
labicmutteringsbegintomeldtogetherintocoherent,oftenusedphrases.
Eventually,onefindshimselfpouringoutentiresentencesandparagraphsof
theUnixMotherTongue,withone’smindentirelyontheproblemathand
(andnotonthesyntaxofanyparticularcommand).Butjustashighschool
foreignlanguagestudentsspendmuchoftheirtimeaskingfordirectionsto
thetoiletandfiguringoutjustwhatthedativecasereallyis,thepathtoLinux
command-line fluency must begin with the first timidly spoken magic words.
Yourshellisveryforgiving,andwillpatiently(andrepeatedly)listentoyour
everyutterance,untilyougetitjustright.Anycommandcanserveasthe
inputforanyother,makingforsomeveryinterestingUnix“sentences.”
Whenarmedwiththehandy(andprobablyover-used)uparrow,itispossi-
bletochaintogethercommandswithslighttweaksovermanytriesto
achieve some very complex behavior.
Forexample,supposethatyou’regiventhetaskoffindingoutwhyaweb
serveristhrowingabunchoferrorsovertime.Ifyoutype lesserror_log,
youseethattherearemany“softerrors”relatingtomissing(orbadly
linked) graphics:
[Tue Aug 27 00:22:38 2002] [error] [client 17.136.12.171] File does not
exist: /htdocs/images/spacer.gif
[Tue Aug 27 00:31:14 2002] [error] [client 95.168.19.34] File does not
exist: /htdocs/image/trans.gifBuilding Complex Command Lines#6
Server Basics | 13
HACK
[Tue Aug 27 00:36:57 2002] [error] [client 2.188.2.75] File does not exist:
/htdocs/images/linux/arrows-linux-back.gif
[Tue Aug 27 00:40:37 2002] [error] [client 2.188.2.75] File does not exist:
/htdocs/images/linux/arrows-linux-back.gif
[Tue Aug 27 00:41:43 2002] [error] [client 6.93.4.85] File does not exist: /
htdocs/images/linux/hub-linux.jpg
[Tue Aug 27 00:41:44 2002] [error] [client 6.93.4.85] File does not exist: /
htdocs/images/xml/hub-xml.jpg
[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not exist: /
htdocs/images/linux/hub-linux.jpg
[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not exist: /
htdocs/images/xml/hub-xml.jpg
andsoon.Runningaloggingpackage(like analog)reportsexactlyhow
manyerrorsyouhaveseeninadaybutfewotherdetails(whichishowyou
wereprobablyalertedtotheprobleminthefirstplace).Lookingatthelog-
filedirectlygivesyoueveryexcruciatingdetailbutisentirelytoomuchinfor-
mation to process effectively.
Let’sstartsimple.Arethere any errorsotherthanmissingfiles?Firstwe’ll
need to know how many errors we’ve had today:
$ wc -l error_log
1265 error_log
And how many were due to File does not exist errors?
$ grep "File does not exist:" error_log | wc -l
1265 error_log
That’sagoodstart.Atleastweknowthatwe’renotseeingpermissionprob-
lemsorerrorsinanythingthatgeneratesdynamiccontent(like cgi scripts.)If
everyerrorisduetomissingfiles(ortyposinourhtmlthatpointtothe
wrongfile)thenit’sprobablynotabigproblem.Let’sgeneratealistofthe
filenames of all bad requests. Hit the up arrow and delete that wc -l:
$ grep "File does not exist:" error_log | awk '{print $13}' | less
That’sthesortofthingthatwewant(the13thfield,justthefilename),but
hangonasecond.Thesamecoupleoffilesarerepeatedmany,manytimes.
Sure,wecouldemailthistothewebteam(allwhopping1265linesofit),
butI’msuretheywouldn’tappreciatetheextraneousspam.Printingeach
file exactly once is easy:
$ grep "File does not exist:" error_log | awk '{print $13}' | sort | uniq |
less
Thisismuchmorereasonable(substitutea wc-l forthat less toseejust
howmanyuniquefileshavebeenlistedasmissing).Butthatstilldoesn’t
reallysolvetheproblem.Maybeoneofthosefileswasrequestedonce,but
anotherwasrequestedseveralhundredtimes.Naturally,ifthereisalink
somewherewithatypoinit,wewouldseemanyrequestsforthesame14 | Server Basics
#6 Building Complex Command Lines
HACK
“missing”file.Butthepreviouslinedoesn’tgiveanyindicationofwhich
filesarerequestedmost.Thisisn’taproblemfor bash;let’stryoutacom-
mand line for loop.
$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
echo -n "$x : "; grep $x error_log | wc -l; done
Weneedthosebackticks(`)toactuallyexecuteourentirecommandfrom
thepreviousexampleandfeedtheoutputofittoa for loop.Oneachitera-
tionthroughtheloop,the$xvariableissettothenextlineofoutputofour
originalcommand(thatis,thenextuniquefilenamereportedasmissing).
Wethen grep forthatfilenameinthe error_log,andcounthowmanytimes
weseeit.The echo attheendjustprintsitinasomewhatnicereportformat.
Icallitasomewhatnicereportbecausenotonlyisitfullofsinglehiterrors
(whichweprobablydon’tcareabout),theoutputisveryjagged,anditisn’t
evensorted!Let’ssortitnumerically,withthebiggesthitsatthetop,num-
bers on the left, and only show the top 20 most requested “missing” files:
$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20
That’s much betterandnotevenmuchmoretypingthanthelasttry.We
needthe tr toeliminatethetrailingnewlineattheendof wc’soutput(whyit
doesn’thaveaswitchtodothis,I’llneverknow).Youroutputshouldlook
something like this:
595 : /htdocs/images/pixel-onlamp.gif.gif
156 : /htdocs/image/trans.gif
139 : /htdocs/images/linux/arrows-linux-back.gif
68 : /htdocs/pub/a/onjava/javacook/images/spacer.gif
50 : /htdocs/javascript/2001/03/23/examples/target.gif
Fromthisreport,it’sverysimpletoseethatalmosthalfofourerrorsaredue
toatypoonapopularwebpagesomewhere(notetherepeated .gif.gif in
thefirstline).Thesecondisprobablyalsoatypo(shouldbe images/,not
image/). The rest are for the web team to figure out:
$ ( echo "Here's a report of the top 20 'missing' files in the error_log.";
echo; \
for x in `grep "File does not exist" error_log | awk '{print $13}' | sort |
uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20 )\
| mail -s "Missing file report" webmaster@oreillynet.com
and maybe one hardcopy for the weekly development meeting:Working with Tricky Files in xargs#7
Server Basics | 15
HACK
$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20 \
| enscript
Hacking the Hack
Onceyougetusedtochunkinggroupsofcommandstogether,youcan
chaintheiroutputstogetherindefinitely,creatinganysortofreportyoulike
outofalivedatastream.Naturally,ifyoufindyourselfdoingaparticular
taskregularly,youmightwanttoconsiderturningitintoashellscriptofits
own(orevenreimplementingitinPerlorPythonforefficiency’ssake,as
every|inacommandmeansthatyou’vespawnedyetanotherprogram).On
amodern(andunloaded)machine,you’llhardlynoticethedifference,but
it’sconsideredgoodformtocleanupthesolutiononceyou’vehackeditout.
And on the command line, there’s plenty of room to hack.
HACK
#7
Working with Tricky Files in xargs Hack #7
Deal with many files containing spaces or other strange characters
Whenyouhaveanumberoffilescontainingspaces,parentheses,andother
“forbidden”characters,dealingwiththemcanbedaunting.Thisisaprob-
lemthatseemstocomeupfrequently,withtherecentexplosivepopularity
ofdigitalmusic.Luckily,tabcompletionin bash makesitsimpletohandle
one file at a time. For example:
rob@catlin:~/Music$ ls
Hallucinogen - The Lone Deranger
Misc - Pure Disco
rob@catlin:~/Music$ rm -rf Misc[TAB]
rob@catlin:~/Music$ rm -rf Misc\ -\ Pure\ Disco/
HittingtheTabkeyfor [TAB] abovereplacesthecommandlinewiththeline
belowit,properlyescapinganyspecialcharacterscontainedinthefile.
That’sfineforonefileatatime,butwhatifwewanttodoamassivetrans-
formation(say,renamingabunchofmp3stoincludeanalbumname)?Take
a look at this:
rob@catlin:~/Music$ cd Hall[TAB]
rob@catlin:~/Music$ cd Hallucinogen\ -\ The\ Lone\ Deranger/
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - 01 - Demention.mp3
Hallucinogen - 02 - Snakey Shaker.mp3
Hallucinogen - 03 - Trancespotter.mp3
Hallucinogen - 04 - Horrorgram.mp3
Hallucinogen - 05 - Snarling (Remix).mp3
Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - 07 - Deranger.mp316 | Server Basics
#7 Working with Tricky Files in xargs
HACK
Hallucinogen - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$
Whenattemptingtomanipulatemanyfilesatonce,thingsgettricky.Many
systemutilitiesbreakonwhitespace(yieldingmanymorechunksthanyou
intended)andwillcompletelyfallapartifyouthrowa ) ora { atthem.What
weneedisadelimiterthatisguaranteednevertoshowupinafilename,and
break on that instead.
Fortunately,the xargs utilitywillbreakonNULLcharacters,ifyouaskitto
nicely. Take a look at this script:
Listing: albumize
#!/bin/sh
if [ -z "$ALBUM" ]; then
echo 'You must set the ALBUM name first (eg. export ALBUM="Greatest Hits")'
exit 1
fi
for x in *; do
echo -n $x; echo -ne '\000'
echo -n `echo $x|cut -f 1 -d '-'`
echo -n " - $ALBUM - "
echo -n `echo $x|cut -f 2- -d '-'`; echo -ne '\000'
done | xargs -0 -n2 mv
We’reactuallydoingtwotrickythingshere.First,we’rebuildingalistcon-
sistingoftheoriginalfilenamefollowedbythenametowhichwe’dliketo
mv it,separatedbyNULLcharacters,forallfilesinthecurrentdirectory.
Wethenfeedthatentirelisttoan xargs withtwoswitches: -0 tellsitto
breakonNULLs(insteadofnewlinesorwhitespace),and -n2 tellsittotake
twoargumentsatatimeoneachpass,andfeedthemtoourcommand(mv).
Savethescriptas ~/bin/albumize.Beforeyourunit,setthe$ALBUMenvi-
ronmentvariabletothenamethatyou’dlikeinjectedintothefilenamejust
after the first -. Here’s a trial run:
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ export ALBUM="The Lone
Deranger"
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ albumize
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - The Lone Deranger - 01 - Demention.mp3
Hallucinogen - The Lone Deranger - 02 - Snakey Shaker.mp3
Hallucinogen - The Lone Deranger - 03 - Trancespotter.mp3
Hallucinogen - The Lone Deranger - 04 - Horrorgram.mp3
Hallucinogen - The Lone Deranger - 05 - Snarling (Remix).mp3
Hallucinogen - The Lone Deranger - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - The Lone Deranger - 07 - Deranger.mp3
Hallucinogen - The Lone Deranger - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$Immutable Files in ext2/ext3#8
Server Basics | 17
HACK
Whatifyouwouldliketoremovethealbumnameagain?Trythisone,and
call it ~/bin/dealbumize:
#!/bin/sh
for x in *; do
echo -n $x; echo -ne '\000'
echo -n `echo $x|cut -f 1 -d '-'`; echo -n ' - '
echo -n `echo $x|cut -f 3- -d '-'`; echo -ne '\000'
done | xargs -0 -n2 mv
and simply run it (no $ALBUM required):
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ dealbumize
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - 01 - Demention.mp3
Hallucinogen - 02 - Snakey Shaker.mp3
Hallucinogen - 03 - Trancespotter.mp3
Hallucinogen - 04 - Horrorgram.mp3
Hallucinogen - 05 - Snarling (Remix).mp3
Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - 07 - Deranger.mp3
Hallucinogen - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$
The -0 switchisalsopopulartoteamupwiththe -print0 optionoffind
(which,naturally,printsmatchingfilenamesseparatedbyNULLsinsteadof
newlines).With find and xargs onapipeline,youcandoanythingyoulike
toanynumberoffiles,withouteverrunningintothedreadedArgumentlist
too long error:
rob@catlin:~/Pit of too many files$ ls *
bash: /bin/ls: Argument list too long
A find/xargs combomakesquickworkofthesefiles,nomatterwhatthey’re
called:
rob@catlin:/Pit of too many files$ find -type f -print0 | xargs -0 ls
To delete them, just replace that trailing ls with an rm, and away you go.
HACK
#8
Immutable Files in ext2/ext3 Hack #8
Create files that even root can’t manipulate
Here’sapuzzle.Supposewe’recleaningup /tmp,andrunintosometrouble:
root@catlin:/tmp# rm -rf junk/
rm: cannot unlink `junk/stubborn.txt': Operation not permitted
rm: cannot remove directory `junk': Directory not empty
root@catlin:/tmp# cd junk/
root@catlin:/tmp/junk# ls -al
total 40
drwxr-xr-x 2 root root 4096 Sep 4 14:45 ./
drwxrwxrwt 13 root root 4096 Sep 4 14:45 ../18 | Server Basics
#8 Immutable Files in ext2/ext3
HACK
-rw-r--r-- 1 root root 29798 Sep 4 14:43 stubborn.txt
root@catlin:/tmp/junk# rm ./stubborn.txt
rm: remove write-protected file `./stubborn.txt'? y
rm: cannot unlink `./stubborn.txt': Operation not permitted
What’sgoingon?Arewerootoraren’twe?Let’stryemptyingthefile
instead of deleting it:
root@catlin:/tmp/junk# cp /dev/null stubborn.txt
cp: cannot create regular file `stubborn.txt': Permission denied
root@catlin:/tmp/junk# > stubborn.txt
bash: stubborn.txt: Permission denied
Well, /tmp certainly isn’t mounted read-only. What is going on?
Inthe ext2 and ext3 filesystems,thereareanumberofadditionalfile
attributesthatareavailablebeyondthestandardbitsaccessiblethrough
chmod.Ifyouhaven’tseenitalready,takealookatthemanpagesfor chattr
and its companion, lsattr.
Oneoftheveryusefulnewattributesis -i,the immutable flag.Withthisbit
set,attemptstounlink,rename,overwrite,orappendtothefileareforbid-
den.Evenmakingahardlinkisdenied(soyoucan’tmakeahardlink,then
editthelink).Andhavingrootprivilegesmakesnodifferencewhenimmuta-
ble is in effect:
root@catlin:/tmp/junk# ln stubborn.txt another.txt
ln: creating hard link `another.txt' to `stubborn.txt': Operation not
permitted
To view the supplementary ext flags that are in force on a file, use lsattr:
root@catlin:/tmp/junk# lsattr
---i--------- ./stubborn.txt
and to set flags a la chmod, use chattr:
root@catlin:/tmp/junk# chattr -i stubborn.txt
root@catlin:/tmp/junk# rm stubborn.txt
root@catlin:/tmp/junk#
Thiscouldbeterriblyusefulforaddinganextrasecuritysteponfilesyou
knowyou’llneverwanttochange(say, /etc/rc.d/* orvariousconfiguration
files.)Whilelittlewillhelpyouonaboxthathasbeenr00ted,immutable
filesprobablyaren’tvulnerabletosimpleoverwriteattacksfromotherpro-
cesses, even if they are owned by root.
Therearehooksforaddingcompression,securitydeletes,undeletability,syn-
chronouswrites,andacoupleofotherusefulattributes.Asofthiswriting,
manyoftheadditionalattributesaren’timplementedyet,butkeepwatching
for new developments on the ext filesystem.Speeding Up Compiles#9
Server Basics | 19
HACK
HACK
#9
Speeding Up Compiles Hack #9
Make sure you’re keeping all processors busy with parallel builds
Ifyou’rerunningamultiprocessorsystem(SMP)withamoderateamountof
RAM,youcanusuallyseesignificantbenefitsbyperformingaparallelmake
whenbuildingcode.Comparedtodoingserialbuildswhenrunningmake
(as is the default), a parallel build is a vast improvement.
Totellmaketoallowmorethanonechildatatimewhilebuilding,use
the -j switch:
rob@mouse:~/linux$ make -j4; make -j4 modules
Someprojectsaren’tdesignedtohandleparallelbuildsandcangetcon-
fusedifpartsoftheprojectarebuiltbeforetheirparentdependencieshave
completed.Ifyourunintobuilderrors,itissafesttojuststartfromscratch
this time without the -j switch.
Bywayofcomparison,herearesomesampletimings.Theywereperformed
onanotherwiseunloadeddualPIII/600with1GBRAM.EachtimeIbuilta
bzImageforLinux2.4.19(redirectingSTDOUTto /dev/null),andremoved
the source tree before starting the next test.
time make bzImage:
real 7m1.640s
user 6m44.710s
sys 0m25.260s
time make -j2 bzImage:
real 3m43.126s
user 6m48.080s
sys 0m26.420s
time make -j4 bzImage:
real 3m37.687s
user 6m44.980s
sys 0m26.350s
time make -j10 bzImage:
real 3m46.060s
user 6m53.970s
sys 0m27.240s
Asyoucansee,thereisasignificantimprovementjustbyaddingthe -j2
switch.Wedroppedfrom7minutesto3minutesand43secondsofactual
time.Increasingto -j4 savedusaboutfivemoreseconds,butjumpingallthe
wayto -j10 actuallyhurtperformancebyafewseconds.Noticehowuserand20 | Server Basics
#10 At Home in Your Shell Environment
HACK
systemsecondsarevirtuallythesameacrossallfourruns.Intheend,youneed
toshovelthesamesizedpileofbits,but -j onamulti-processormachinesim-
ply lets you spread it around to more people with shovels.
Ofcourse,bitsalleventuallyendupinthebitbucketanyway.Buthey,if
nothing else, performance timings are a great way to keep your cage warm.
HACK
#10
At Home in Your Shell Environment Hack #10
Make bash more comfortable through environment variables
Consultingamanpagefor bash canbeadauntingread,especiallyifyou’re
notpreciselysurewhatyou’relookingfor.Butwhenyouhavethetimeto
devotetoit,themanpagefor bash iswellworththeread.Thisisashelljust
oozingwithallsortsofarcane(butwonderfullyuseful)features,mostof
which are simply disabled by default.
Let’sstartbylookingatsomeusefulenvironmentvariables,andsomeuse-
ful values to which to set them:
export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`
Asyouprobablyknow,thePS1variablesetsthedefaultsystemprompt,and
automaticallyinterpretsescapesequencessuchas\u(forusername)and\w
(forthecurrentworkingdirectory.)Asyoumaynotknow,itispossibleto
encodeANSIescapesequencesinyourshellprompt,togiveyourprompta
colorizedappearance.Wewrapthewholestringinbackticks(`)inorderto
get echo togeneratethemagicASCIIescapecharacter.Thisisexecuted
once,andtheresultisstoredinPS1.Let’slookatthatlineagain,withbold-
face around everything that isn’t an ANSI code:
export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`
Youshouldrecognizethefamiliar\u@\h:\w\$promptthatwe’veallgrownto
knowandlove.Bychangingthenumbersjustaftereachsemicolon,youcan
set the colors of each part of the prompt to your heart’s content.
Alongthesamelines,here’sahandycommandthatisrunjustbefore bash
gives you a prompt:
export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
(Wedon’tneedbackticksforthisone,as bash isexpectingittocontainan
actualcommand,notastring.)Thistime,theescapesequenceisthemagic
stringthatmanipulatesthetitlebaronmostterminalwindows(suchas
xterm,rxvt,eterm,gnometerm,etc.).Anythingafterthesemicolonand
beforethe\007getsprintedtoyourtitlebareverytimeyougetanew
prompt.Inthiscase,we’redisplayingyourusername,thehostyou’relogged
into,andthecurrentworkingdirectory.Thisisquitehandyforbeingableto
Download from Wow! eBook <www.wowebook.com>At Home in Your Shell Environment#10
Server Basics | 21
HACK
tellataglance(orevenwhilewithin vim)towhichmachineyou’reloggedin,
andtowhatdirectoryyou’reabouttosaveyourfile.See“ConstantLoad
AverageDisplayintheTitlebar” [Hack#59] ifyou’dliketoupdateyourtitlebar
in real time instead of at every new bash prompt.
Haveyoueveraccidentallyhit^Dtoomanytimesinarow,onlytofind
yourselfloggedout?Youcantell bash toignoreasmanyconsecutive^D
hits as you like:
export IGNOREEOF=2
Thismakes bash followtheSnarkrule(“WhatItellyouthreetimesistrue”)
andonlylogyououtifyouhit^Dthreetimesinarow.Ifthat’stoofewfor
you, feel free to set it to 101 and bash will obligingly keep count for you.
Havingadirectoryjustoffofyourhomethatliesinyourpathcanbe
extremelyuseful(forkeepingscripts,symlinks,andotherrandompiecesof
code.)Atraditionalplacetokeepthisdirectoryisin bin underneathyour
home directory. If you use the ~ expansion facility in bash, like this:
export PATH=$PATH:~/bin
thenthepathwillalwaysbesetproperly,evenifyourhomedirectoryever
getsmoved(orifyoudecideyouwanttousethissamelineonmultiple
machineswithpotentiallydifferenthomedirectories—asin movein.sh).See
“Get Settled in Quickly with movein.sh” [Hack #72].
DidyouknowthatjustascommandsaresearchedforinthePATHvariable
(andmanpagesaresearchedforintheMANPATHvariable),directoriesare
likewisesearchedforintheCDPATHvariableeverytimeyouissuea cd?By
default, it is only set to “.”, but can be set to anything you like:
export CDPATH=.:~
Thiswillmake cd searchnotonlythecurrentdirectory,butalsoyourhome
directory for the directory you try to change to. For example:
rob@caligula:~$ ls
bin/ devel/ incoming/ mail/ test/ stuff.txt
rob@caligula:~$ cd /usr/local/bin
rob@caligula:/usr/local/bin$ cd mail
bash: cd: mail: No such file or directory
rob@caligula:/usr/local/bin$ export CDPATH=.:~
rob@caligula:/usr/local/bin$ cd mail
/home/rob/mail
rob@caligula:~/mail$
YoucanputasmanypathsasyouliketosearchforinCDPATH,separating
each with a : (just as with the PATH and MANPATH variables.)22 | Server Basics
#10 At Home in Your Shell Environment
HACK
Weallknowabouttheuparrowandthe history command.Butwhathap-
pensifyouaccidentallytypesomethingsensitiveonthecommandline?Sup-
poseyouslipwhiletypingandaccidentallytypeapasswordwhereyou
meanttotypeacommand.Thisaccidentwillfaithfullygetrecordedtoyour
~/.bash_history filewhenyoulogout,whereanotherunscrupuloususer
mighthappentofindit.Editingyour .bash_history manuallywon’tfixthe
problem, as the file gets rewritten each time you log out.
To clear out your history quickly, try this from the command line:
export HISTSIZE=0
Thiscompletelyclearsoutthecurrentbashhistoryandwillwriteanempty
.bash_history onlogout.Whenyoulogbackin,yourhistorywillstartover
fromscratchbutwillotherwiseworkjustasbefore.Fromnowon,trytobe
more careful!
Doyouhaveaproblemwithpeopleloggingintoamachine,thendiscon-
nectingtheirlaptopandgoinghomewithoutloggingbackoutagain?If
you’veeverrunawandseenabunchofidleuserswhohavebeenloggedin
for several days, try setting this in their environment:
export TMOUT=600
TheTMOUTvariablespecifiesthenumberofsecondsthat bash willwaitfor
inputonacommandlinebeforeloggingtheuseroutautomatically.This
won’thelpifyourusersaresittingina vi windowbutwillalleviatetheprob-
lemofusersjustsittingatanidleshell.Tenminutesmightbealittleshort
forsomeusers,butkindlyremindthemthatiftheydon’tlikethesystem
default, they are free to reset the variable themselves.
Thisbringsupaninterestingpoint:exactlywheredoyougotomakeanyof
theseenvironmentchangespermanent?Thereareseveralfilesthat bash con-
sultswhenstartingup,dependingonwhethertheshellwascalledatloginor
from within another shell.
From bash(1), on login shells:
...itfirstreadsandexecutescommandsfromthefile /etc/profile,ifthatfile
exists.Afterreadingthatfile,itlooksfor ~/.bash_profile, ~/.bash_login,and
~/.profile,inthatorder,andreadsandexecutescommandsfromthefirstone
thatexistsandisreadable.... Whenaloginshellexits, bash readsandexe-
cutes commands from the file ~/.bash_logout, if it exists.
For all other shells:
bash reads and executes commands from ~/.bashrc, if that file exists.
Forthefulldefinitionofwhatconstitutesaloginshell(andforawhole
bunchofinformationabouttheenvironmentyouworkineveryday),con-
sult bash(1).Finding and Eliminating setuid/setgid Binaries#11
Server Basics | 23
HACK
HACK
#11
FindingandEliminatingsetuid/setgidBinariesHack #11
Eliminate potential root exploits before they have a chance to happen
WhilerunningLinuxasaserver,oneguidingprinciplethathasservedme
wellistocontinuallyask,“whatamItryingtoachieve?”Doesitmakesense
forawebservertohavetheprintingsubsysteminstalled?Shouldasystem
withnoconsolehave gpm installed?Usually,extrasoftwarepackagesjust
takeupunnecessarydiskspace,butinthecaseofsetuidorsetgidbinaries,
the situation could be far worse.
Whiledistributionmaintainersworkveryhardtoensurethatallknown
exploitsforsetuidandsetgidbinarieshavebeenremoved,itseemsthata
fewnewunexpectedexploitscomeouteverymonthortwo.Especiallyif
yourserverhasmoreshellusersthanyourself,youshouldregularlyaudit
thesetuidandsetgidbinariesonyoursystem.Chancesareyou’llbesur-
prised at just how many you’ll find.
Here’s one command for finding all the files with a setuid or setgid bit set:
root@catlin:~# find / -perm +6000 -type f -exec ls -ld {} \; > setuid.txt &
Thiswillcreateafilecalled setuid.txt thatcontainsthedetailsofallofthe
matchingfilespresentonyoursystem.Itisaverygoodideatolookthrough
this list, and remove the s bits of any tools that you don’t use.
Let’s look through what we might find on a typical system:
-rws--x--x 1 root bin 35248 May 30 2001 /usr/bin/at
-rws--x--x 1 root bin 10592 May 30 2001 /usr/bin/crontab
Notmuchsurprisehere. at and crontab needrootprivilegesinorderto
changetotheuserthatrequestedthe at jobor cron job.Ifyou’reparanoid,
andyoudon’tusethesefacilities,thenyoucouldremovethesetuidbits
with:
# chmod a-s /usr/bin/{at,crontab}
Generallyspeaking,it’sabadideatodisable cron (assomanysystems
dependontimedjobexecution).Butwhenwasthelasttimeyouused at?
Doyourusersevenknowwhatit’sfor?Personally,Ifind at aniceshortcut
tosettingupafull-blown cron job,andwouldn’tliketopartwithit.Butif
thereisnocallforitonyourparticularsystem,youshouldconsider
defangingit.Withthesetuidbitremoved,thecommandswillnolongerbe
available to regular users but will still work fine as root.
-rws--x--x 1 root bin 11244 Apr 15 2001 /usr/bin/disable-paste
Thisispartofthe gpm package(amousedriverfortheLinuxconsole).Do
youhaveamouseattachedtotheconsoleofthismachine?Doyouuseitin24 | Server Basics
#11 Finding and Eliminating setuid/setgid Binaries
HACK
textmode?Ifnot,thenwhyleaveasetuidrootbinaryinplacethatwillnever
even be called?
-r-s--s--x 1 root lp 14632 Jun 18 2001 /usr/bin/lpq
-r-s--s--x 1 root lp 15788 Jun 18 2001 /usr/bin/lpr
-r-s--s--x 1 root lp 15456 Jun 18 2001 /usr/bin/lprm
-r-xr-s--x 1 root lp 23772 Jun 18 2001 /usr/sbin/lpc
Theseareallpartoftheprintingsubsystem.Doesthismachineactuallyuse
lp to print?
-rws--x--x 1 root bin 33760 Jun 18 2000 /usr/bin/chage
-rws--x--x 1 root bin 29572 Jun 18 2000 /usr/bin/chfn
-rws--x--x 1 root bin 27188 Jun 18 2000 /usr/bin/chsh
-rws--x--x 1 root bin 35620 Jun 18 2000 /usr/bin/passwd
Theseareallnecessaryforuserstobeabletosettheirpasswords,shell,and
fingerinformation.Doesyoursiteuse finger informationatall?If,likemost
sites,youhavearostersomewhereelse(probablyontheweb)thatisn’tkept
insyncwithuser’sGECOSfield,thenthisinformationisgenerallyuseless
(exceptfortheuser’s“real”name,whichisstillusedinsomeemailclients).
Doyoureallyneedtoallowuserstochangethisinformationontheirown,
without admin intervention?
-r-xr-sr-x 1 root tty 9768 Jun 21 2001 /usr/bin/wall
-r-xr-sr-x 1 root tty 8504 Jun 21 2001 /usr/bin/write
Bothwallandwriteneedtobesetgid tty towritetootheruser’sterminals.
Thisisgenerallyasafeoperation,butcanbeabusedbymiscreantswholike
towritebaddata(or lots ofbaddata)tootheruser’sterminals.Ifyoudon’t
needtoprovidethisfunctionalitytootherusers,whynotdisablethesetgid
bit?Ifyoudo,rootwillstillbeabletosendwallsandwrites(forexample,
when a message is sent by shutdown when rebooting the system).
-rwsr-xr-x 1 root bin 14204 Jun 3 2001 /usr/bin/rcp
-rwsr-xr-x 1 root bin 10524 Jun 3 2001 /usr/bin/rlogin
-r-sr-xr-x 1 root bin 7956 Jun 3 2001 /usr/bin/rsh
The r* commandsareleftfromanage(perhapsnotsolongago),beforethe
daysof ssh.Doyouneedtoprovidethe r commandstoyourusers?Isthere
anythingthat ssh and scp can’tdothatyouabsolutelyneed rsh and rcp for?
Morethanlikely,onceyou’veworkedwith ssh forawhile,you’llnevermiss
the r commands,andcansafelyremovethepotentialtickingtimebombofa
disusedsetuid rsh installation.Ifyou’relookingforinterestingthingstodo
with ssh, check out any of the ssh hacks elsewhere in this book.
-r-sr-xr-x 1 root bin 10200 Jun 3 2001 /usr/bin/traceroute
-r-sr-xr-x 1 root bin 15004 Jun 3 2001 /bin/ping
The traceroute and ping commandsneedthesetuidrootbittobeabletocre-
ateICMPpackets.IfyouwantyouruserstobeabletorunthesenetworkMake sudo Work Harder#12
Server Basics | 25
HACK
diagnostictools,thenthey’llneedtobesetuidroot.Otherwise,removeset-
uid and then only root will be able to run ping and traceroute.
-r-sr-sr-- 1 uucp uucp 83344 Feb 10 2001 /usr/bin/uucp
-r-sr-sr-- 1 uucp uucp 36172 Feb 10 2001 /usr/bin/uuname
-r-sr-sr-- 1 uucp uucp 93532 Feb 10 2001 /usr/bin/uustat
-r-sr-sr-- 1 uucp uucp 85348 Feb 10 2001 /usr/bin/uux
-r-sr-sr-- 1 uucp uucp 65492 Feb 10 2001 /usr/lib/uucp/uuchk
-r-sr-sr-- 1 uucp uucp 213832 Feb 10 2001 /usr/lib/uucp/uucico
-r-sr-sr-- 1 uucp uucp 70748 Feb 10 2001 /usr/lib/uucp/uuconv
-r-sr-sr-- 1 uucp uucp 315 Nov 22 1995 /usr/lib/uucp/uusched
-r-sr-sr-- 1 uucp uucp 95420 Feb 10 2001 /usr/lib/uucp/uuxqt
WhenwasthelasttimeyouconnectedtoanothermachinewithUUCP?
HaveyoueversetuptheUUCPsystem?Ihavebeenanetworkadminfor
tenyears,andinthattime,IhavenevercomeacrossaliveUUCPinstalla-
tion.That’snottosaythatUUCPisn’tuseful,justthatinthesedaysofper-
manentlyconnectedTCP/IPnetworks,UUCPisbecomingextremely
uncommon.Ifyou’renotusingUUCP,thenleavingsetuidandsetgidbina-
ries online to support it doesn’t make much sense.
Doanyofthebinariesintheexamplesabovehavepotentialroot(orother
privilegeelevation)exploits?Ihavenoidea.ButIdoknowthatbyremoving
unnecessaryprivileges,Iminimizemyexposuretothepossibilitythatan
exploit might be run on this system if one is discovered.
Ihavetriedtopresentthishackasaprocess,notasasolution.Thislistis
certainlybynomeansdefinitive.Asyoubuildaserver,alwayskeepinmind
whattheintendeduseis,andbuildthesystemaccordingly.Wheneverpossi-
ble,removeprivileges(orevenentirepackages)thatprovidefunctionality
thatyousimplydon’tneed.Consultthemanpageifyoueverwonderabout
whataparticularbinaryissupposedtobedoing,andwhyitisinstalledset-
uid (and when the manpage fails you, remember to use the source).
HACK
#12
Make sudo Work Harder Hack #12
Usesudotoletotherusersdoyourevilbidding,withoutgivingawaythe
machine
The sudo utilitycanhelpyoudelegatesomesystemresponsibilitiestoother
people,withoutgivingawayfullrootaccess.Itisasetuidrootbinarythat
executescommandsonanauthorizeduser’sbehalf,aftertheyhaveentered
their current password.
Asroot,run /usr/sbin/visudo toeditthelistofuserswhocancall sudo.The
default sudo list looks something like this:
root ALL=(ALL) ALL26 | Server Basics
#12 Make sudo Work Harder
HACK
Unfortunately,manysystemadminstendtousethisentryasatemplateand
grant unrestricted root access to all other admins unilaterally:
root ALL=(ALL) ALL
rob ALL=(ALL) ALL
jim ALL=(ALL) ALL
david ALL=(ALL) ALL
Whilethismayallowyoutogiveoutrootaccesswithoutgivingawaythe
rootpassword,itisreallyonlyausefulmethodwhenallofthe sudo users
canbecompletelytrusted.Whenproperlyconfigured,the sudo utilityallows
fortremendousflexibilityforgrantingaccesstoanynumberofcommands,
run as any arbitrary uid.
The syntax of the sudo line is:
user machine=(effective user) command
Thefirstcolumnspecifiesthe sudo user.Thenextcolumndefinesthehosts
inwhichthissudoentryisvalid.Thisallowsyoutoeasilyuseasingle sudo
configuration across multiple machines.
Forexample,supposeyouhaveadeveloperwhoneedsrootaccessona
development machine, but not on any other server:
peter beta.oreillynet.com=(ALL) ALL
Thenextcolumn(inparentheses)specifiestheeffectiveuserthatmayrun
thecommands.Thisisveryhandyforallowinguserstoexecutecodeas
users other than root:
peter lists.oreillynet.com=(mailman) ALL
Finally, the last column specifies all the commands that this user may run:
david ns.oreillynet.com=(bind) /usr/sbin/rndc,/usr/sbin/named
Ifyoufindyourselfspecifyinglargelistsofcommands(or,forthatmatter,
usersormachines),thentakeadvantageof sudo’sAliassyntax.AnAliascan
be used in place of its respective entry on any line of the sudo configuration:
User_Alias ADMINS=rob,jim,david
User_Alias WEBMASTERS=peter,nancy
Runas_Alias DAEMONS=bind,www,smmsp,ircd
Host_Alias WEBSERVERS=www.oreillynet.com,www.oreilly.com,www.perl.com
Cmnd_Alias PROCS=/bin/kill,/bin/killall,/usr/bin/skill,/usr/bin/top
Cmnd_Alias APACHE=/usr/local/apache/bin/apachectl
WEBMASTERS WEBSERVERS=(www) APACHE
ADMINS ALL=(DAEMONS) ALLUsing a Makefile to Automate Admin Tasks#13
Server Basics | 27
HACK
Itisalsopossibletospecifysystemgroupsinplaceoftheuserspecification
toallowanyuserwhobelongstothatgrouptoexecutecommands.Just
preface the group with a %, like this:
%wwwadmin WEBSERVERS=(www) APACHE
Nowanyuserwhoispartofthewwwadmingroupcanexecute apachectl as
the www user on any of the web server machines.
OneveryusefulfeatureistheNOPASSWD: flag.Whenpresent,theuser
won’t have to enter his password before executing the command:
rob ALL=(ALL) NOPASSWD: PROCS
Thiswillallowtheuserrobtoexecute kill, killall, skill,and top onany
machine, as any user, without entering a password.
Finally, sudo canbeahandyalternativeto su forrunningcommandsatstar-
tup out of the system rc files:
(cd /usr/local/mysql; sudo -u mysql ./bin/safe_mysqld &)
sudo -u www /usr/local/apache/bin/apachectl start
Forthattoworkatboottime,you’llneedthedefaultline rootALL=(ALL)
ALL to be present.
Use sudo withtheusualcaveatsthatapplytosetuidbinaries.Particularlyif
youallow sudo toexecuteinteractivecommands(likeeditors)oranysortof
compilerorinterpreter,youshouldassumethatitispossiblethatthe sudo
userwillbeabletoexecutearbitrarycommandsastheeffectiveuser.Still,
undermostcircumstancesthisisn’taproblemandiscertainlypreferableto
giving away undue access to root privileges.
HACK
#13
Using a Makeﬁle to Automate Admin Tasks Hack #13
Makefiles make everything (not just gcc) faster and easier
Youprobablyknowthe make commandfrombuildingprojects(probably
involving gcc)fromsource.Butnotmanypeoplealsorealizethatsinceit
keepstrackoffilemodificationtimes,itcanbeahandytoolformakingall
sorts of updates whenever arbitrary files are updated.
Here’s a Makefile that is used to maintain sendmail configuration files.
Listing: Makeﬁle.mail
M4= m4
CFDIR= /usr/src/sendmail-8.12.5/cf
CHMOD= chmod
ROMODE= 44428 | Server Basics
#13 Using a Makefile to Automate Admin Tasks
HACK
RM= rm -f
.SUFFIXES: .mc .cf
all: virtusers.db aliases.db access.db sendmail.cf
access.db: access.txt
makemap -v hash access < access.txt
aliases.db: aliases
newaliases
virtusers.db: virtusers.txt
makemap -v hash virtusers < virtusers.txt
.mc.cf:
$(RM) $@
$(M4) ${CFDIR}/m4/cf.m4 $*.mc > $@ || ( $(RM) $@ && exit 1 )
$(CHMOD) $(ROMODE) $@
Withthisinstalledas /etc/mail/Makefile,you’llneverhavetorememberto
run newaliases wheneditingyoursendmailaliasesfile,orthesyntaxof
that makemap commandwhenyouupdatevirtualdomainoraccesscon-
trolsettings.Andbestofall,whenyouupdateyourmaster mc
configurationfile(youareusing mc andnoteditingthe sendmail.cf by
hand,right?)thenitwillbuildyournew .cf fileforyou—allbysimplytyp-
ing make.Sincemakekeepstrackoffilesthathavebeenrecentlyupdated,it
takes care of rebuilding only what needs to be rebuilt.
Here’sanotherexample,usedtopushApacheconfigurationfilesto
anotherserver(say,in around-robinApachesetup,whichyoucanlearn
moreaboutin“DistributingLoadwithApacheRewriteMap” [Hack#99].Just
put this in your /usr/local/apache/conf directory:
Listing: Makeﬁle.push
#
# Makefile to push *.conf to the slave, as needed.
#
SLAVE= www2.oreillynet.com
APACHE= /usr/local/apache
RM= /bin/rm
TOUCH= /bin/touch
SSH= /usr/local/bin/ssh
SCP= /usr/local/bin/scp
.SUFFIXES: .conf .ts
all: test restart sites.ts globals.ts httpd.tsBrute Forcing Your New Domain Name#14
Server Basics | 29
HACK
configtest: test
test:
@echo -n "Testing Apache configuration: "
@$(APACHE)/bin/apachectl configtest
restart:
$(APACHE)/bin/apachectl restart
.conf.ts:
@$(RM) -f $@
@$(SCP) $*.conf $(SLAVE):$(APACHE)/conf
@$(SSH) $(SLAVE) $(APACHE)/bin/apachectl restart
@$(TOUCH) $@
Thisexampleisalittletrickierbecausewe’renotactuallybuildinganynew
files,soit’sdifficultfor make totellifanyoftheconfigurationfileshaveactu-
allychanged.Wefakeout make bycreatingempty .ts files(shortforTimeS-
tamp)thatonlyservetoholdthetimeanddateofthelastupdate.Ifthereal
files(httpd.conf, sites.conf,or globals.conf)havechanged,thenwefirstrunan
apachectlconfigtest toverifythatwehaveagoodconfiguration.Ifallgoeswell,
itwillthenrestartthelocalApache,copythenewlychangedfilesovertothe
slaveserver,thenrestartApacheontheslaveserver.Finally,wetouchtherele-
vant .ts files so we won’t process them again until the .conf files change.
Thissavesalotoftypingandissignificantlyquicker(andsafer)thandoing
it all by hand on each update.
HACK
#14
Brute Forcing Your New Domain Name Hack #14
Find exactly the domain you’d like to register, whatever it turns out to be
Therearemanytoolsavailableonlinethatwillassistinperformingwhois
queriesforyoutodetermineifyourfavoritedomainnameisstillavailable,
andifnot,whohasregisteredit.Thesetoolsareusuallywebbasedand
allowyoutosubmitafewqueriesatatime(andfrequentlysuggestseveral
inane alternatives if your first choice is taken).
Ifyou’renotsomuchinterestedinaparticularnameasinfindingonethat
matchesapattern,whynotletthecommandlinedotheworkforyou?Sup-
pose you wanted to find a list of all words that end in the letters “st”:
cat /usr/share/dict/words | grep 'st$' | sed 's/st$/.st/' | \
while read i; do \
(whois $i | grep -q '^No entries found') && echo $i; sleep 60; \
done | tee list_of_st_domains.txt
Thiswillobliginglysupplyyouwithavisualrunningtabofallavailable
wordsthathaven’tyetbeenregisteredtoelRepublicaDemocraticadeSào30 | Server Basics
#15 Playing Hunt the Disk Hog
HACK
ToméePríncipe(thedomainregistrarforthestTLD).Thisexample
searchesthesystemdictionaryandtriestofindthewhoisrecordforeach
matchingword,oneatatime,every60seconds.Itsavesanynonexistent
recordstoafilecalled list_of_st_domains.txt,andshowsyouitsprogressas
itruns.ReplacethatstwithanytwoletterTLD(likeusorto)tobruteforce
the namespace of any TLD you like.
SomefeelthatthedomainnamelandgrabisalmostturningtheInternetinto
acorporateghetto,butIdon’tsubscribetothatidea.Iactuallyfindthe
whole situation quite humorous.
HACK
#15
Playing Hunt the Disk Hog Hack #15
Browse your filesystem for heavy usage quickly with a handy alias
ItalwaysseemstohappenlateonaSaturdaynight.You’regettingpaged
becauseapartitionononeoftheservers(probablythemailserver)isdan-
gerously close to full.
Obviously, running a df will show what’s left:
rob@magic:~$ df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda1 7040696 1813680 4863600 27% /
/dev/sda2 17496684 13197760 3410132 79% /home
/dev/sdb1 8388608 8360723 27885 100% /var/spool/mail
Butyoualreadyknewthatthemailspoolwasfull(hence,thepagethattook
youawayfromanotherwisepleasant,non-mailserverrelatedevening).How
can you quickly find out who’s hogging all of the space?
Here’s a one-liner that’s handy to have in your .profile:
alias ducks='du -cks * |sort -rn |head -11'
Oncethisaliasisinplace,running ducks inanydirectorywillshowyouthe
totalinuse,followedbythetop10diskhogs,indescendingorder.It
recursessubdirectories,whichisveryhandy(butcantakealongtimetorun
onaheavilyloadedserver,orinadirectorywithmanysubdirectoriesand
files in it). Let’s get to the bottom of this:
rob@magic:~$ cd /var/spool/mail
rob@magic:/var/spool/mail$ ducks
8388608 total
1537216 rob
55120 phil
48800 raw
43175 hagbard
36804 mal
30439 erisFun with /proc#16
Server Basics | 31
HACK
30212 ferris
26042 nick
22464 rachael
22412 valis
Oops!Itlookslikemymailspoolrunnethover.Boy,Ihaveordersof
magnitudemoremailthananyotheruser.I’dbetterdosomethingabout
that,suchasappropriatenewhardwareandupgradethe /var/spool/mail
partition. ;)
Asthiscommandrecursessubdirectories,it’salsogoodforrunningaperi-
odic report on home directory usage:
root@magic:/home# ducks
[ several seconds later ]
13197880 total
2266480 ferris
1877064 valis
1692660 hagbard
1338992 raw
1137024 nick
1001576 rob
925620 phil
870552 shared
607740 mal
564628 eris
Forrunningsimplespotcheckswhilelookingfordiskhogs, ducks cansave
manykeystrokes(althoughifwecalleditsomethinglike ds,itwouldsave
even more, but wouldn’t be nearly as funny).
HACK
#16
Fun with /proc Hack #16
Directly view the kernel’s running process table and system variables
The /proc filesystemcontainsarepresentationofthekernel’sliveprocesstable.
Bymanipulatingfilesanddirectoriesin /proc,youcanlearnabout(andfiddle
with)allsortsofparametersintherunningsystem.Bewarnedthatpoking
aroundunder /proc asrootcanbeextraordinarilydangerous,asroothasthe
powertooverwritevirtuallyanythingintheprocesstable.Oneslipofaredi-
rector,andLinuxwillobliginglyblowawayyourentirekernelmemory,with-
out so much as a “so long and thanks for all the kcore.”
Herearesomeexamplesofinterestingthingstodowith /proc.Inthese
examples,we’llassumethatyou’reusingarecentkernel(about2.4.18orso)
andthatyouareloggedinasroot.Unlessyou’reroot,youwillonlybeable
to view and modify processes owned by your uid.32 | Server Basics
#16 Fun with /proc
HACK
First, let’s take a look at a lightly loaded machine:
root@catlin:/proc# ls
1/ 204/ 227/ 37/ bus/ hermes/ loadavg scsi/ version
1039/ 212/ 228/ 4/ cmdline ide/ locks self@
1064/ 217/ 229/ 5/ cpuinfo interrupts meminfo slabinfo
1078/ 220/ 230/ 6/ devices iomem misc stat
194/ 222/ 231/ 698/ dma ioports modules swaps
197/ 223/ 232/ 7/ driver/ irq/ mounts sys/
2/ 224/ 233/ 826/ execdomains kcore net/ sysvipc/
200/ 225/ 254/ 827/ filesystems kmsg partitions tty/
202/ 226/ 3/ apm fs/ ksyms pci uptime
Thedirectoriesconsistingofnumberscontaininformationabouteverypro-
cessrunningonthesystem.ThenumbercorrespondstothePID.Therestof
thefilesanddirectoriescorrespondtodrivers,counters,andmanyother
internalsoftherunningkernel.Theinterfaceoperatesjustasanyotherfile
ordeviceinthesystem,byreadingfromandwritingtoeachentryasifit
were a file. Suppose you want to find out which kernel is currently booted:
root@catlin:/proc# cat version
Linux version 2.4.18 (root@catlin) (gcc version 2.95.3 20010315 (release))
#2 Sat Jun 22 19:01:17 PDT 2002
Naturally,youcouldfindmuchofthatoutbysimplyrunning uname-a,
butthiswaycutsoutthemiddleman(andactuallydoeswhat uname does
internally).
Interested in how much RAM we have installed? Take a look at kcore:
root@catlin:/proc# ls -l kcore
-r-------- 1 root root 201330688 Aug 28 21:39 kcore
Lookslikewehave192MBinstalledinthismachine(201330688/1024/
1024==192,moreorless).Noticetherestrictedfilepermissions?Thatis
thesystem’sdefenseagainstanyoneattemptingtoreadthememory
directly.Ofcourse,you’rerootandcandowhateveryoulike.There’s
nothingpreventingyoufromrunning grep or strings onkcoreandlooking
forinterestingtidbits.Thisisthesystemmemory,andthingsgetcachedin
thereuntiloverwritten(makingitpossibletohuntdownaccidentallylost
dataortrackdownnaughtypeopledoingthingstheyoughtn’t).Grovel-
lingoverkcoreisactuallynotmuchfunandistypicallyonlyamethod
used by the very desperate (or very bored).
Some other notable status files:
root@catlin:/proc# cat interrupts
CPU0
0: 34302408 XT-PIC timer
1: 2 XT-PIC keyboard
2: 0 XT-PIC cascade
3: 289891 XT-PIC orinoco_cs
Download from Wow! eBook <www.wowebook.com>Fun with /proc#16
Server Basics | 33
HACK
8: 1 XT-PIC rtc
9: 13933886 XT-PIC eth0
10: 25581 XT-PIC BusLogic BT-958
14: 301982 XT-PIC ide0
NMI: 0
ERR: 0
Thesearethesystemcountersforeveryinterruptthathaseverbeencalled,
its number, and the driver that called it.
root@catlin:/proc# cat partitions
major minor #blocks name
8 0 8971292 sda
8 1 8707198 sda1
8 2 257040 sda2
3 64 29316672 hdb
3 65 29310561 hdb1
Thisisalistofalloftheharddiskpartitions(anddevices)thatwerediscov-
eredatboot,alongwiththeirrespectivesizes.Herewehavea9GBSCSI
disk,anda30GBIDEdisk.Allavailablepartitionsarerepresentedhere,
regardlessofwhethertheyarecurrentlymounted(makingitahandyrefer-
ence to see if there are any unmounted disks on the system).
Let’sleavethesystemparametersandtakealookatthestructureofanindi-
vidual process.
root@catlin:/proc# cd 1
root@catlin:/proc/1# ls -l
total 0
-r--r--r-- 1 root root 0 Aug 28 22:05 cmdline
lrwxrwxrwx 1 root root 0 Aug 28 22:05 cwd -> //
-r-------- 1 root root 0 Aug 28 22:05 environ
lrwxrwxrwx 1 root root 0 Aug 28 22:05 exe -> /sbin/init*
dr-x------ 2 root root 0 Aug 28 22:05 fd/
-r--r--r-- 1 root root 0 Aug 28 22:05 maps
-rw------- 1 root root 0 Aug 28 22:05 mem
lrwxrwxrwx 1 root root 0 Aug 28 22:05 root -> //
-r--r--r-- 1 root root 0 Aug 28 22:05 stat
-r--r--r-- 1 root root 0 Aug 28 22:05 statm
-r--r--r-- 1 root root 0 Aug 28 22:05 status
Therearethreeinterestingsymlinksinthisdirectory. cwd pointstothecur-
rentworkingdirectoryofthisprocess(youcan,forexample, cd/proc/3852/
cwd tolandinthedirectoryfromwhichprocessID3852wasrun.)The exe
linkpointstofullpathtothebinarythatwascalled,and root pointstothe
notionoftherootdirectorythatthisprocesshas.The root linkwillalmost
always be /, unless it has executed a chroot.
The cmdline and environ filescontainthecommandlineasitwasoriginally
calledandtheprocessescompleteenvironment.TheseareseparatedbyNULL
characters, so to see them in a more human readable form, try this:34 | Server Basics
#17 Manipulating Processes Symbolically withprocps
HACK
root@catlin:/proc/1# cat environ |tr '\0' '\n'
HOME=/
TERM=linux
BOOT_IMAGE=catlin
(orforabetterexample,trythison /proc/self/environ,theenvironmentof
the currently running process):
root@catlin:/proc/1# cat /proc/self/environ |tr '\0' '\n'
PWD=/proc/1
HOSTNAME=catlin.nocat.net
MOZILLA_HOME=/usr/lib/netscape
ignoreeof=10
LS_OPTIONS= --color=auto -F -b -T 0
MANPATH=/usr/local/man:/usr/man:/usr/X11R6/man
LESSOPEN=|lesspipe.sh %s
PS1=\u@\h:\w\$
PS2=>
...
andsoon.Thiscanbetremendouslyhandyforuseinshellscripts(orother
programs)whereyouneedspecificinformationaboutrunningprocesses.
Justuseitwithcare,andrememberthatunprivilegeduserscanusuallyonly
access information about their own processes.
Inclosing,here’sapracticalexampleofoneusefor /proc.Bycheckingthe
output of ps against the running process table, you can see if the two agree:
# ls -d /proc/* |grep [0-9]|wc -l; ps ax |wc -l
Thiswillgiveyouaquickspotcheckofthenumberofrunningprocesses
versusthenumberthat ps actuallyreports.Manyrootkitsinstallahacked ps
thatallowsamiscreanttohideprocesses(bysimplynotdisplayingthem
when ps runs).Youmayhidefrom ps,butit’smuchmoredifficulttohide
from /proc.Ifthesecondnumberisconsiderablylargerthanthefirst(partic-
ularlyifyourunitseveraltimesinarow)thenyoumightwanttoconsider
taking your box offline for further inspection. Quickly.
HACK
#17
Manipulating Processes Symbolically
withprocps Hack #17
Signal and renice processes by name, terminal, or username (instead of PID)
Ifyouoftenfindyourselfrunninga psawux|grepsomething justtofindthe
PIDofajobyou’dliketokill,thenyoushouldtakealookatsomeofthe
more modern process manipulation packages.
Probablythebestknownprocesstoolspackageis procps,thesamepackage
thatincludestheLinuxversionoftheubiquitous top command.The top tool
issotremendouslyusefulandflexiblethatitdeservesitsowndiscussion.
Learn more about it in“Monitor System Resources with top” [Hack #58].Manipulating Processes Symbolically withprocps#17
Server Basics | 35
HACK
Amongtheotherniftyutilitiesincludedin procps: skill letsyousendsignals
toprocessesbyname,terminal,username,orPID,and snice doesthesame
but renices processes instead of sending them signals.
For example, to freeze the user on terminal pts/2:
# skill -STOP pts/2
To release them from the grip of sleeping death, try this:
# skill -CONT pts/2
Or to renice all of luser’s processes to 5:
# snice +5 luser
pkill issimilarto skill,butwithmoreformalparameters.Ratherthan
attemptingtoguesswhetheryouarereferringtoausername,processname,
orterminal,youspecifythemexplicitlywithswitches.Forexample,these
two commands do the same thing:
# skill -KILL rob bash
# pkill -KILL -u rob bash
pkill maytakeslightlymoretyping,butisguaranteedtobeunambiguous
(assumingthatyouhappenedtohaveauserandaprocesswiththesame
name, for example).
pgrep worksjustlike pkill,butinsteadofsendingasignaltoeachprocess,it
simply prints the matching PID(s) on STDOUT:
$ pgrep httpd
3211
3212
3213
3214
3215
3216
Finally, vmstat givesyouanice,easilyparsedpinpointmeasurementofvir-
tual memory and cpu statistics:
$ vmstat
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 5676 6716 35804 58940 0 0 9 9 7 9 0 0 29
Ifyou’dliketowatchhowusagechangesovertime,giveitanumberonthe
commandline.Thenumberrepresentsthenumberofsecondstopause
before displaying the results of another measurement.
Learninghowtousethelesserknown procps utilitiescansaveyoulotsof
typing,nottomentionwincing.Use skill or pkill toavoidaccidentally
mistypingaPIDargumentto kill,andsuddenlybringing sshd (andthewrath
of many angry users) down upon your unfortunate sysadmin head.36 | Server Basics
#18 Managing System Resources per Process
HACK
See also:
• ftp://people.redhat.com/johnsonm/procps/
•“Monitor System Resources with top” [Hack #58]
•manpages for pkill, pgrep, skill, snice, and vmstat
HACK
#18
Managing System Resources per Process Hack #18
Prevent user processes from running away with all system resources
Whetherintentionallyoraccidentally,itisentirelypossibleforasingleuser
touseupallsystemresources,leadingtopoorperformanceoroutrightsys-
temfailure.Onefrequentlyoverlookedwaytodealwithresourcehogsisto
use the ulimit functionality of bash.
Topreventaprocess(oranyofitschildren)fromcreatingenormousfiles,
try specifying a ulimit -f (with the maximum file size specified in kilobytes).
rob@catlin:/tmp$ ulimit -f 100
rob@catlin:/tmp$ yes 'Spam spam spam spam SPAM!' > spam.txt
File size limit exceeded
rob@catlin:/tmp$ ls -l spam.txt
-rw-r--r-- 1 rob users 102400 Sep 4 17:05 spam.txt
rob@catlin:/tmp$
Userscandecreasetheirownlimits,butnotincreasethem(aswith nice and
renice).Thismeansthatulimitssetin /etc/profile cannotbeincreasedlater
by users other than root:
rob@catlin:/tmp$ ulimit -f unlimited
bash: ulimit: cannot modify limit: Operation not permitted
Notethatnothingispreventingauserfromcreatingmanyfiles,allasbigas
theirulimitallows.Userswiththisparticulartemperamentshouldbe
escortedtoabackroomandintroducedtoyourfavoriteLART.Oralter-
nately,youcouldlookintointroducingdiskquotas(althoughthisisusually
less than fun to implement, if a simple stern talking to will fix the problem).
Likewise, ulimit canlimitthemaximumnumberofchildrenthatasingle
user can spawn:
rob@catlin:~$ cat > lots-o-procs
#!/bin/bash
export RUN=$((RUN + 1))
echo $RUN...
$0
^D
rob@catlin:~$ ulimit -u 10
rob@catlin:~$ ./lots-o-procs
1...
2...Managing System Resources per Process#18
Server Basics | 37
HACK
3...
4...
5...
6...
7...
8...
9...
./lots-o-procs: fork: Resource temporarily unavailable
rob@catlin:~$
Thislimitsthenumberofprocessesforasingleuseracrossallterminals(and
backgroundedprocesses).Ithastobethisway,becauseonceaprocessis
forked,itdisassociatesitselffromthecontrollingterminal.(Andhowwould
you count it against a given subshell then?)
Oneotherveryuseful ulimit optionis -v,maximumvirtualmemorysize.
Oncethisceilingisreached,processeswillexitwithSegmentationfault
(whichisn’tideal,butwillkeepthesystemfromcrashingasitrunsoutof
RAMandswap).Ifyouhaveaparticularlybadlybehavingprocessthat
showssignificantbloat(likeApache+mod_perlandpoorlywrittenCGI
code,forexample)youcouldsetaulimittoactasan“emergencybrake”
whiledebuggingtherealsourceofthetrouble.Again,specifythelimitin
kilobytes.
To see all available ulimit settings, use -a:
rob@catlin:~$ ulimit -a
core file size (blocks) 0
data seg size (kbytes) unlimited
file size (blocks) unlimited
max locked memory (kbytes) unlimited
max memory size (kbytes) unlimited
open files 1024
pipe size (512 bytes) 8
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 1536
virtual memory (kbytes) unlimited
Youcanseethatbeforesettingsystem-widehardlimits,userprocessescan
grow to be quite large. In tcsh, the analogous command you’re after is limit:
rob@catlin:~> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 1536
openfiles 102438 | Server Basics
#19 Cleaning Up after Ex-Users
HACK
Settingsystemresourcelimitsmaysounddraconianbutisamuchbetter
alternative to the downward spiral of a user process gone amok.
See also:
• http://www.tuxedo.org/~esr/jargon/html/entry/LART.html
HACK
#19
Cleaning Up after Ex-Users Hack #19
Make sure you close the door all the way when a user takes their leave
Ithappens.Planschange,companiesshiftfocus,andpeoplemoveon.At
somepoint,everyadminhashadtocleanupshellaccessaftersomeonehas
left the company, happily or otherwise.
Iampersonallyoftheopinionthatifonedoesn’ttrustone’susersfromthe
beginning,thentheywillonedayprovethemselvesuntrustworthy.(Of
course,I’veneverhadtoadminanopenshellserveratanISP,either.)Atany
rate,buildingtrustwithyourusersfromthebeginningwillgoalongway
toward being able to sleep at night later on.
Whenyoudohavetolockoldaccountsup,it’sbesttoproceedstrategi-
cally.Don’tassumethatjustbecauseyourana passwd-l thattheuserin
questioncan’tregainaccesstoyourmachine.Let’sassumethatwe’relock-
ingupafteranaccountcalled luser.Herearesomeobvious(andsomenotso
obvious) things to check on in the course of cleaning up:
passwd -l luser
Obviously, locking the user’s password is a good first step.
chsh -s /bin/true luser
Thisisanotherpopularstep,changingtheuser’sloginshelltosomething
thatexitsimmediately.Thisgenerallypreventsauserfromgainingshell
accesstotheserver.Butbewarned,if sshd isrunningonthisbox,andyou
allowremoteRSAorDSAkeyauthentication,then luser canstillforward
ports to any machine that your server can reach! With a command like this:
luser@evil:~$ ssh -f -N -L8000:private.intranet.server.com:80 old.server.com
luser hasjustforwardedhislocalport8000toyourinternalintranetserver’s
httpport.Thisisallowedsince luser isn’tusingapassword(heisusingan
RSAkey)andisn’tattemptingtoexecuteaprogramonold.server.com
(since he specified the -N switch).
Obviously,youshouldremove ~luser/.ssh/authorized_keys* andprevent luser
from using his ssh key in the first place. Likewise, look for either of these files:
~luser/.shosts
~luser/.rhostsCleaning Up after Ex-Users#19
Server Basics | 39
HACK
Thisusuallyisn’taproblemunlessyou’rerunning rsh,orhaveenabledthis
functionalityin ssh.Butyouneverknowifafutureadminwilldecideto
enable .shosts or .rhosts functionality,soit’sbettertoremovethemnow,if
they exist.
Did luser have any sudo privileges? Check visudo to be sure.
How about cron jobs or at jobs?
crontab -u luser -e
atq
For that matter, is luser running any jobs right now?
ps awux |grep -i ^luser
orasin“ManipulatingProcessesSymbolicallywithprocps” [Hack#17],you
can use:
skill -KILL luser
Could luser executecgiprogramsfromhishomedirectory(orsomewhere
else)?
find ~luser/public_html/ -perm +111
What about PHP or other embedded scripting languages?
find ~luser ~public_html/ -name '*.php*'
Does luser haveanyemailforwardingsetup?Forwarderscanfrequentlybe
made to execute arbitrary programs.
less ~luser/.forward
grep -C luser /etc/mail/aliases
Finally, does luser own any files in strange places?
find / -user luser > ~root/luser-files.report
Onesafe(andquick)wayofensuringthatallof luser’spersonalconfigura-
tionfilesareinvalidatedisto mv/home/luser/home/luser.removed.Thiswill
keepthecontentsof luser’shomedirectoryintact,withoutworryingabout
havingmissedotherpossiblepointsofentry.Notethatthiswillbreaka
legitimate .forward file(andalso ~luser/public_html andanyotherpublicly
accessibledatathat luser mighthavekeptinhishomedirectory),soifyou
gothisroutebesuretotakethatintoaccount(say,byaddinganappropri-
ateentrytothesystem aliases file,andmovingacleanedversionof public_
html/ back to /home/luser/public_html/.
Lookatconfigurationfilesforanysystemsoftwaretowhich luser had
access,particularlyservicesthatrunasprivilegedusers.Evensomethingas
innocuousasauser-suppliedApacheconfigurationfilecouldbeusedtopro-
vide shell access later.40 | Server Basics
#20 Eliminating Unnecessary Drivers from the Kernel
HACK
Thislistisbynomeansexhaustivebutismeanttodemonstratethatthereis
alotmoretorevokingaccessthansimplylockingauser’spassword.Ifthis
usereverhadrootaccess,allbetsareoff.Accesscouldlaterbegrantedby
anythingfromaTrojansystembinarytoaninvisiblekernelmoduletohav-
ing simply changed the root password.
Gettoknowyourshelluserslongbeforethedayyouhavetosaygoodbye.
Thisjobisdifficultenoughwithouthavingtoworryaboutwhomyoucan
trust.
HACK
#20
Eliminating Unnecessary Drivers from the
Kernel Hack #20
Keep your kernel optimized for quick booting and long-term stability
Linuxwillrunonanenormousvarietyofcomputerhardware.Thereissup-
portforallmannerofhardware,fromcriticalcomponents(suchasharddisk
drivesandRAM)tomoreexoticdevices(suchasUSBscannersandvideo
captureboards).ThekernelthatshipswithmostLinuxdistributionsaimsto
becomplete(andsafe)attheexpenseofpossiblybeinglessefficientthanit
could be, by including support for as many devices as possible.
Asyourmachineboots,takealookatthemessagesthekernelproduces.
Youmayfinditprobingforallsortsofhardware(particularlySCSIcontrol-
lersandEthernetcards)thatyoudon’tactuallyhaveinstalled.Ifyourdistri-
butionhidesthekernelbootmessages,trythe dmesg command(probably
piped through less) to see what kernel messages were generated at boot.
Tomakeyourkernelbootquicklyandatthesametimeeliminatethepossi-
bilitythatanunnecessarydevicedrivermightbecausingproblemswithyour
installedhardware,youshouldtrimdownthedriversthatthekernel
attempts to load to fit your hardware.
Therearetwoschoolsofthoughtonmanagingkerneldrivers.Somepeople
prefertobuildakernelwithallofthefunctionalitytheyneedbuiltintoit,
withoutusingloadablekernelmodules.Othersprefertobuildamorelight-
weightkernelandloadthedriverstheyneedwhenthesystemboots.Of
course, both methods have their advantages and disadvantages.
Forexample,themonolithickernel(withoutloadablemodules)isguaran-
teedtoboot,evenifsomethinghappenstothedriversunder /lib/modules.
Someadminsevenprefertobuildakernelwithnoloadablemodulesupport
atall,todiscouragethepossibilityofTrojanhorsedevicedriversbeing
loadedbyarandommiscreantdowntheroad.Ontheotherhand,ifanew
pieceofhardwareisaddedtothesystem,thenyouwillneedtorebuildyour
kernel to accommodate it.Eliminating Unnecessary Drivers from the Kernel#20
Server Basics | 41
HACK
Ifyouuseloadablemodules,thenyouhaveenormousflexibilityinhowyou
loaddevicedrivers.Youcanaltertheorderthatdriversloadandevenpass
parameterstovariousmodulesasyouloadthedriver,allwithoutrebooting.
Thedownsideisthatgoodcopiesofthemodulesmustexistunder /lib/
modules,orelsethesystemcan’tloaditsdrivers.Whenyoubuildanewker-
nel,you’llneedtoremembertoruna makemodules_install,andtokeep
yourkernelmoduleutilities(like modprobe and lsmod)uptodate.Building
akernelwithloadablemodulescanalsohelpifyourmonolithickernelistoo
largetoboot(byloadingextradevicedriversafterthekernelbootsandthe
filesystems are mounted).
Regardlessofthemethodthatyouchooseforyoursystem,you’llneedto
buildakernelwiththeminimalfunctionalitythatisrequiredtoboot.This
includesdriversfortheIDE,SCSI,orotherbus(maybefloppydiskoreven
networkcard?)thatyourmachinebootsfrom.Youwillalsoneedsupport
forthefilesystemthatyourrootpartitionisinstalledon(likely ext2, ext3,or
reiserfs).Makesurethatyoubuildakernelwithsupportfortheamountof
RAMthatyourmachinehasinstalled(see“UsingLargeAmountsofRAM”
[Hack#21]).Selectaprocessorthatmatchesyourhardwaretobesurethatall
appropriateoptimizationsareturnedon.BesuretobuildanSMPkernelif
you have more than one processor.
Tobuildacustomizedkernel,unpackthekernelsourcessomewherethat
hasenoughroom(say,in /usr/local/src/linux).Runa makemenuconfig (or,if
you’rerunningX, makexconfig).Selectyourdriverscarefully(hittingYfor
built-indrivers,andMforloadablemodules.)Rememberthatyou’rebuild-
ingakernelforaserveranddon’tincludeextraneousdrivers.Doesyour
serverreallyneedsoundsupport,evenifitisbuiltintoyourmotherboard?
WhataboutUSB?Unlessyouhaveaspecificuseforaparticularpieceof
hardwarethatisdirectlyrelatedtoitsjobasaserver,don’tbotherinstalling
adriverforit.Youshouldalsoconsiderdisablingunusedhardwareinthe
BIOS,whereverpossible,toconservesystemresourcesandreducethepossi-
bility of hardware resource conflicts.
Afteryou’refinishedselectingwhichbitsofthekerneltobuild,itwillsave
yourconfigurationtoafilecalled .config atthetopofthekernelsourcetree.
Savethisfileforlater,becauseitwillmakeupgradingyourkernelmucheas-
ierdowntheroad(bycopyingittothetopofthenewkerneltree,andrun-
ning makeoldconfig).Nowbuildthenewkernel(andmodules,if
applicable),installit,andgiveitatry.Besuretotryoutallofyourdevices
afterinstallinganewkernel,andwatchyourbootmessagescarefully.Track
downanyunexpectedwarningsorerrorsnow,beforetheycausetroubleat
some future date.42 | Server Basics
#21 Using Large Amounts of RAM
HACK
Finally,ifyou’redoingkernelworkremotely,don’tforgettobuildinsup-
portforyournetworkdevices!There’snothingquitelikethepainofrealiz-
ingthatthenetworkdriversaren’tbuiltjust after youissuea shutdown-r
now.Hopefullythatconsoleisaccessiblebysomeoneoncall(whowilldis-
cretelybootyouroldkernelforyou,withouttelling too manyofyour
friends).
Buildingakernelwell-tailoredtoyourhardwarecanbechallenging,butis
necessarytomakeyourserverrunasefficientlyasitcan.Don’tbediscour-
agedifittakesacoupleoftriestobuildtheperfectLinuxkernel,theeffortis
well worth it.
See also:
• Running Linux, Fourth Edition (O’Reilly)
•“Using Large Amounts of RAM” [Hack #21]
HACK
#21
Using Large Amounts of RAM Hack #21
Be sure that Linux is using all of your available system RAM
Linuxiscapableofaddressingupto64GBofphysicalRAMonx86sys-
tems.Butifyouwanttoaccommodatemorethan960MBRAM,you’llhave
to let the system know about it.
Firstofall,yourLinuxkernelmustbeconfiguredtosupporttheadditional
RAM.Typically,thedefaultkernelconfigurationwilladdressupto960MB
RAM.Ifyouinstallmorethanthatinamachine,itwillsimplybeignored.
(Thecommoncomplaintisthatyou’vejustinstalled1GB,andyeta‘free’
only reports 960MB, even though it counts to 1024 MB at post time.)
Thewaythatthekerneladdressesitsavailablesystemmemoryisdictated
bytheHighMemorySupportsetting(a.k.a.theCONFIG_NOHIGH-
MEMdefine.)DependingontheamountofRAMyouintendtouse,setit
accordingly:
up to 960MB: off
up to 4GB: 4GB
more than 4GB: 64GB
Bewarnedthatselecting64GBrequiresaprocessorcapableofusingIntel
PhysicalAddressExtension(PAE)mode.Accordingtothekernelnotes,all
IntelprocessorssincethePentiumProsupportPAE,butthissettingwon’t
workonolderprocessors(andthekernelwillrefusetoboot,whichisone
reasonthatitisn’tonbydefault).Makeyourselectionandrebuildyourker-
nel, as in“Eliminating Unnecessary Drivers from the Kernel” [Hack #20].hdparm: Fine Tune IDE Drive Parameters#22
Server Basics | 43
HACK
Oncethekernelisbuiltandinstalled,youmayhavetotellyourbootloader
howmuchRAMisinstalled,soitcaninformthekernelatboottime(asnot
everyBIOSisaccurateinreportingthetotalsystemRAMatboot.)Todo
this,addthe mem= kernelparameterinyourbootloaderconfiguration.For
example, suppose we have a machine with 2GB RAM installed.
If you’re using Lilo, add this line to /etc/lilo.conf:
append="mem=2048M"
If you’re using Grub, try this in your /etc/grub.conf:
kernel /boot/vmlinuz-2.4.19 mem=2048M
If you’re running loadlin, just pass it on the loadlin line:
c:\loadlin c:\kernel\vmlinuz root=/dev/hda3 ro mem=2048M
Although,ifyou’rerunning loadlin,whyareyoureadingabookonLinux
Server Hacks? ;)
HACK
#22
hdparm: Fine Tune IDE Drive Parameters Hack #22
Get the best possible performance from your IDE hardware
Ifyou’rerunningaLinuxsystemwithatleastone(E)IDEharddrive,and
you’ve never heard of hdparm, read on.
Bydefault,mostLinuxdistributionsusethedefaultkernelparameterswhen
accessingyourIDEcontrolleranddrives.Thesesettingsareveryconserva-
tiveandaredesignedtoprotectyourdataatallcosts.Butasmanyhave
cometodiscover,safealmostneverequalsfast.Andwithlargevolumedata
processing applications, there is no such thing as “fast enough.”
IfyouwanttogetthemostperformanceoutofyourIDEhardware,takea
lookatthe hdparm(8) command.Itwillnotonlytellyouhowyourdrivesare
currently performing, but will let you tweak them to your heart’s content.
Itisworthpointingoutthatundersomecircumstances,thesecommands
CANCAUSEUNEXPECTEDDATACORRUPTION!Usethematyour
ownrisk!Attheveryleast,backupyourboxandbringitdowntosingle-
user mode before proceeding.
Let’sbegin.Nowthatwe’reinsingleusermode(whichwediscussedin
“ForgoingtheConsoleLogin” [Hack#2]),let’sfindouthowwelltheprimary
drive is currently performing:
hdparm -Tt /dev/hda
You should see something like:
/dev/hda:
Timing buffer-cache reads: 128 MB in 1.34 seconds =95.52 MB/sec
Timing buffered disk reads: 64 MB in 17.86 seconds = 3.58 MB/sec44 | Server Basics
#22 hdparm: Fine Tune IDE Drive Parameters
HACK
Whatdoesthistellus?The -T meanstotestthecachesystem(i.e.,themem-
ory,CPU,andbuffercache).The -t meanstoreportstatsonthediskin
question,readingdatanotinthecache.Thetwotogether,runacoupleof
timesinarowinsingle-usermode,willgiveyouanideaoftheperformance
ofyourdiskI/Osystem.(TheseareactualnumbersfromaPII/350/128M
Ram/EIDE HD; your numbers will vary.)
Butevenwithvaryingnumbers,3.58MB/secispatheticfortheabovehard-
ware.IthoughttheadfortheHDsaidsomethingabout66MBpersec-
ond!!?!? What gives?
Let’s find out more about how Linux is addressing this drive:
# hdparm /dev/hda
/dev/hda:
multcount = 0 (off)
I/O support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 0 (off)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 1870/255/63, sectors = 30043440, start = 0
Thesearethedefaults.Nice,safe,butnotnecessarilyoptimal.What’sall
this about 16-bit mode? I thought that went out with the 386!
Thesesettingsarevirtuallyguaranteedtoworkonanyhardwareyoumight
throwatit.Butsinceweknowwe’rethrowingsomethingmorethanadusty,
8-year-old, 16-bit multi-IO card at it, let’s talk about the interesting options:
multcount
Shortformultiplesectorcount.Thiscontrolshowmanysectorsare
fetchedfromthediskinasingleI/Ointerrupt.AlmostallmodernIDE
drives support this. The manpage claims:
Whenthisfeatureisenabled,ittypicallyreducesoperatingsystemoverhead
fordiskI/Oby30-50%.Onmanysystems,italsoprovidesincreaseddata
throughput of anywhere from 5% to 50%.
I/O support
Thisisabigone.ThisflagcontrolshowdataispassedfromthePCIbus
tothecontroller.Almostallmoderncontrollerchipsetssupportmode3,
or32-bitmodew/sync.Someevensupport32-bitasync.Turningthis
on will almost certainly double your throughput (see below).
unmaskirq
TurningthisonwillallowLinuxtounmaskotherinterruptswhilepro-
cessingadiskinterrupt.Whatdoesthatmean?ItletsLinuxattendtohdparm: Fine Tune IDE Drive Parameters#22
Server Basics | 45
HACK
otherinterrupt-relatedtasks(i.e.,networktraffic)whilewaitingforyour
disktoreturnwiththedataitaskedfor.Itshouldimproveoverallsys-
temresponsetime,butbewarned:notallhardwareconfigurationswill
be able to handle it. See the manpage.
using_dma
DMAcanbeatrickybusiness.Ifyoucangetyourcontrolleranddrive
usingaDMAmode,doit.However,Ihaveseenmorethanonemachine
hang while playing with this option. Again, see the manpage.
Let’s try out some turbo settings:
# hdparm -c3 -m16 /dev/hda
/dev/hda:
setting 32-bit I/O support flag to 3
setting multcount to 16
multcount = 16 (on)
I/O support = 3 (32-bit w/sync)
Great!32-bitsoundsnice.Andsomemulti-readsmightwork.Let’sre-run
the benchmark:
# hdparm -tT /dev/hda
/dev/hda:
Timing buffer-cache reads: 128 MB in 1.41 seconds =90.78 MB/sec
Timing buffered disk reads: 64 MB in 9.84 seconds = 6.50 MB/sec
Hmm, almost double the disk throughput without really trying! Incredible.
Butwait,there’smore:we’restillnotunmaskinginterrupts,usingDMA,or
evenausingdecentPIOmode!Ofcourse,enablingthesegetsriskier.The
manpage mentions trying Multiword DMA mode2, so let’s try this:
# hdparm -X34 -d1 -u1 /dev/hda
Unfortunatelythisseemstobeunsupportedonthisparticularbox(ithung
likeanNTboxrunningaJavaapplication)So,afterrebootingit(againin
single-user mode), I went with this:
# hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda
/dev/hda:
setting 32-bit I/O support flag to 3
setting multcount to 16
setting unmaskirq to 1 (on)
setting using_dma to 1 (on)
setting xfermode to 66 (UltraDMA mode2)
multcount = 16 (on)
I/O support = 3 (32-bit w/sync)
unmaskirq = 1 (on)
using_dma = 1 (on)
Download from Wow! eBook <www.wowebook.com>46 | Server Basics
#22 hdparm: Fine Tune IDE Drive Parameters
HACK
And then checked:
# hdparm -tT /dev/hda
/dev/hda:
Timing buffer-cache reads: 128 MB in 1.43 seconds =89.51 MB/sec
Timing buffered disk reads: 64 MB in 3.18 seconds =20.13 MB/sec
20.13 MB/sec. A far cry from the miniscule 3.58 with which we started.
Didyounoticehowwespecifiedthe -m16 and -c3 switchagain?That’s
becauseitdoesn’trememberyour hdparm settingsbetweenreboots.Besure
toaddtheabovelinetoyour /etc/rc.d/* scriptsonceyou’resurethesystemis
stable(andpreferablyafteryour fsck runs;runninganextensivefilesystem
checkwithyourcontrollerinaflakymodemaybeagoodwaytogenerate
vastquantitiesofentropy,butit’snowaytoadministerasystem.Atleast
not with a straight face).
Ifyoucan’tfind hdparm onyoursystem(usuallyin /sbin or /usr/sbin),getit
from the source at http://metalab.unc.edu/pub/Linux/system/hardware/.47
Chapter2 CHAPTER TWO
Revision Control
Hacks #23–36
Ifyoutakeadministrationseriously,youwilllikelyspenduntoldhoursfine-
tuningyoursystemtobehave justso.Unfortunately,itisn’talwayspossible
todefinewhenagivenapplicationis“working”andwhenitis“broken”;
usuallyfunctionalityismuchmorefinelygraded,somewhereinthegrey
areabetweenupanddown.Evenmakingasmallconfigurationchangecan
bringaboutsubtleeffectsthataren’tseenfordays(orweeks)downtheroad.
ThisiswhereRevisionControlcanbeapowerfulally.Beyondproviding
simplebackups,anyrevisioncontrolpackageworthitsweightinbitswill
keeptrackofthechangesinanyfile,whenitwaschanged,whochangedit,
andwhythechangewasmade.Thisinformationcanbeofuntoldvalue
whenworkingonacomplexsystem,particularlyifmorethanonepersonis
responsibleforitsupkeep.Bykeepingcriticalconfigurationfilesinsystems
suchasRCSorCVS,youwillbeableto“rollback”toanypreviousrevision
ofafile,andanalyzethedifferencesbetweenwhatwaspresentthenandthe
version you have online right now.
Inthissection,thegoalistoprovidethespecificsyntaxyouneedtomake
effectiveuseofRCSandCVS,withusefulexamplesofhowyoumightuse
them.WhileRCSisextremelyhandyformaintainingfilerevisionsonasingle
machine,CVSoffersextensivearchivingandmergingcapabilities,keepinga
centralrepositorysomewhereonthenetwork.Understandably,CVSisa
muchmorecomplicatedtoolthanRCS.Hacks23through25offeracrash
courseinRCS,while26through36willgetyouuptospeedquicklyonCVS,
from simple commands to setting up your own anonymous CVS repository.
Naturally,thischapterassumesthatyou’realreadycomfortableworking
withfilesfromthecommandline,evenifyouhaven’tworkedwithRCSor
CVSbefore.Ifyou’relookingforin-depthinformationaboutCVS,takea
lookattheexcellentbook OpenSourceDevelopmentwithCVS byKarlFogel
(CoriolisOpen Press).48 | Revision Control
#23 Getting Started with RCS
HACK
HACK
#23
Getting Started with RCS Hack #23
Let RCS manage your system files, and keep a revision history
RCSisarevisioncontrolsystem,usefulforkeepingmultiplerevisionsofcon-
figurationfiles,shellscripts,andanyothertextfilesyouhavelyingaround.
UnlikeCVS,thereisnofunctionalityforusingRCSwithremotemachines;in
RCS, everything is intended to be kept in the local filesystem.
RCSkeepsallitsrevisionsinadirectorycalled RCS/ (underthedirectory
you’re currently in). To start a new RCS repository, just make the directory:
root@catlin:/etc# mkdir RCS
Now,we’llneedtoinitializeafileintothenewRCSrepositorybeforework-
ing with it. Let’s work with syslog.conf:
root@catlin:/etc# ci -i syslog.conf
RCS/syslog.conf,v <-- syslog.conf
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Here's the syslog.conf for Catlin
initial revision: 1.1
done
Theinitial ci-i doesa“checkinandinitialize,”togiveRCSagoodfirstcopy
ofthefile.RCSpromptsforaninitialdescription,andthenremovesthefile
fromthecurrentdirectory.Youprobablydon’twanttoleaveitlikethatfor
too long, so after you’ve initialized the file in RCS, check it out:
root@catlin:/etc# co syslog.conf
RCS/syslog.conf,v --> syslog.conf
revision 1.1
done
Thatcopiesthefilebackintothecurrentdirectory.Butwe’renotreadyto
startworkingonitjustyet.First,let’scheckitoutwithalocktoprevent
other people from making updates while we work on it.
root@catlin:/etc# co -l syslog.conf
RCS/syslog.conf,v --> syslog.conf
revision 1.1 (locked)
done
Nowyoucaneditthefiletoyourheart’scontent.Onceyou’refinishedwith
it, check it back in and unlock it with ci -u:
root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
new revision: 1.3; previous revision: 1.2
enter log message, terminated with single '.' or end of file:
>> Added remote logging to the new security log host
>> .
doneChecking Out a Previous Revision in RCS#24
Revision Control | 49
HACK
Besuretousemeaningfullogentries,becauseyou won’t rememberwhat
youmeantby“madeacoupleofedits”insixmonths,whenyourcodestarts
doing strange things.
Ifyouhavetroublecheckinganythinginorout,you’veprobablyforgotten
the-l(onco)or-u(onci)switchatsomepoint.Youcanusuallystartover
bymakingabackupcopy,thencheckingthefilebackoutagain,andcopy-
ing it back, assuming that you’ve just edited syslog.conf:
root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
ci: syslog.conf,v: no lock set by root
root@catlin:/etc#
Uh-oh. Better back it up and start again from scratch:
root@catlin:/etc# cp syslog.conf syslog.conf.backup
root@catlin:/etc# co -l syslog.conf
syslog.conf,v --> syslog.conf
revision 1.4 (locked)
done
root@catlin:/etc# cp syslog.conf.backup syslog.conf
root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
new revision: 1.5; previous revision: 1.4
enter log message, terminated with single '.' or end of file:
>> commented out the FIFO line
>> .
done
We’ll see some more fun things to do with RCS in the next couple of hacks.
HACK
#24
Checking Out a Previous Revision in RCS Hack #24
Save yourself with RCS revisions
Inevitably,youwillbreakthingssohorriblythatyouhavenohopeofget-
tingthembackagain.Supposeyou’vejustdoneacomplicatededittoyour
httpd.conf,andyougettheerrormessagethatstrikesfear(andoccasionally
heartburn) in the heart of admins everywhere:
root@catlin:/usr/local/apache/conf# ../bin/apachectl restart
apachectl restart: httpd not running, trying to start
apachectl restart: httpd could not be started
Oh,dear.Whathaveyoudone?Withthewebserverdown,peoplearen’t
evengettingyourfriendly404page,they’reseeingConnectionRefused
errors.Youcoulddiginto httpd.conf desperatelywith vi tofigureoutexactly
what you (or perhaps the admin before you) did to deserve this.
Butifyou’reusingRCS,thingsaren’tsobleak.Toseewhathaschanged
between your last checkout and this one, use rcsdiff:50 | Revision Control
#25 Tracking Changes with rcs2log
HACK
root@catlin:/usr/local/apache/conf# rcsdiff httpd.conf
===================================================================
RCS file: RCS/httpd.conf,v
retrieving revision 1.1
diff -r1.1 httpd.conf
458c458
< ErrorLog /usr/local/apache/logs/error_log
---
> ErrorLog :wq/usr/local/apache/logs/error_log
Thereweare.Evidently,a :wq accidentallygotinsertedatline458.Ah,must
havemissedthatESCkey.Makeyourfix,startApache,andcheckitbackin.
Butwhatifthechangesweremoreextensivethanthat?Supposeyoumakea
bigchangetoaconfigurationfileonTuesdaymorning,andgremlinsstartto
manifestthemselvesaroundThursdayafternoon.Howcanyougetbacktoa
known good copy?
Simple: check out a particular revision using the -r switch:
root@catlin:/etc# co -l -r1.2 syslog.conf
syslog.conf,v --> syslog.conf
revision 1.2
done
Presto,you’rebacktothestateyouwereinwhenyoucheckedthefileinat
rev 1.2. Remember with RCS, it makes sense to save early and often.
HACK
#25
Tracking Changes with rcs2log Hack #25
See at a glance who’s been editing your files, and why
RCSreallyshowsitsstrengthwhenmorethanonepersonneedstoedita
particularfile.Havingmultipleadminscansometimesleadtoablamegame
whenthingsaren’tworkingproperly.Inthesecases,it’sobviousthatsome-
bodychanged something,butnobodywilladmittoit.Ifyoukeeptrackof
changes in RCS, then it’s simple to tell who’s been changing what:
root@www:/usr/local/apache/htdocs# rcs2log index.html
2002-08-14 rob <rob@mouse>
* index.html: meeting announcement
2002-07-30 rob <rob@mouse>
* index.html: gre.tunnel announcement
2002-07-12 sderle <sderle@mouse>
* index.html: added Marin and shuffled around a bit.
2002-07-10 rob <rob@mouse>Tracking Changes with rcs2log#25
Revision Control | 51
HACK
* index.html: meeting announcement + v0.81
2002-07-01 jim <jim@mouse>
* index.html: *** empty log message ***
2002-06-20 rob <rob@mouse>
* index.html: meeting reminder
Hmm,what’sthatemptylogmessagedoingthere?Itmightbeworthtalk-
ingtoJim(andpossiblydoingan rcsdiff betweenitandthepreviousrevi-
sion).Buthowcanyoutellwhichversionscorrespondtowhichlogentries?
Try the -v switch on rcs2log:
root@www:/usr/local/apache/htdocs# rcs2log -v index.html
2002-08-14 rob <rob@mouse>
* index.html 1.54: meeting announcement
2002-07-30 rob <rob@mouse>
* index.html 1.53: gre.tunnel announcement
2002-07-12 sderle <sderle@mouse>
* index.html 1.52: added Marin and shuffled around a bit.
2002-07-10 rob <rob@mouse>
* index.html 1.51: meeting announcement + v0.81
2002-07-01 jim <jim@mouse>
* index.html 1.50: *** empty log message ***
2002-06-20 rob <rob@mouse>
* index.html 1.49: meeting reminder
Now do a diff between whichever revisions you like:
root@www:/usr/local/apache/htdocs# rcsdiff -r1.49 -r1.50 index.html
===================================================================
RCS file: RCS/index.html,v
retrieving revision 1.49
retrieving revision 1.50
diff -r1.49 -r1.50
199a200,202
> <dt><a href="http://labs.google.com/">Google Labs</a></dt>
> <dd>Take a look at the strange and wondrous things that Google is up to...
</dd>
>52 | Revision Control
#26 Getting Started with CVS
HACK
Soitwasjustalinkaddition.Still,itmightbeeasytojusthitthat^Dwhen
prompted, but it’s generally considered bad form to not make any log entry.
Atanyrate,onceyoumasterthesimple ci, co, rcsdiff,and rcs2log com-
mands,youhavesomeveryflexiblerevisioncontroltoolsatyourdisposal.
Usethemwell,andthey’llprobablysaveyourlifeoneday.Orinthevery
least, they’ll save the next best thing: your data.
HACK
#26
Getting Started with CVS Hack #26
The ins and outs of the Concurrent Versioning System
ConcurrentVersioningSystem(CVS)isasystemformanagingsimulta-
neousdevelopmentoffiles.Itiscommonlyusedinlargeprogramming
projectsandisalsousefultosystemadministrators,technicalwriters,and
anyone who needs to manage files.
CVSstoresfilesinacentralrepository.Itissetuptobeaccessibletoall
usersofthefiles,usingstandardUnixpermissions.Commandsaregivento
“checkout”acopyofafilefordevelopmentand“commit”changesbackto
therepository.Italsoscansthefilesastheyaremovedtoandfromthe
repository, to prevent one person’s work from overwriting another’s.
Thissystemensuresthatahistoryofthefileisretained,whichisextremely
usefulwhenthebossdecideshewantsafeatureyoutrashedmonthsago.It
alsoensuresthatbackinguptherepositoryisenoughtobackupaproject
(providing all necessary files are kept in repository).
Typical Uses
CVSisdesignedfordevelopers,eitherindividuallyorinteams.Forindividu-
als,CVSprovidesarepositoryfromwhichyoucanworkfromhome,the
office,ortheclientsitewithouthavingtohauldisksaround.Italsopro-
videsversioncontrol,allowingrollbackswithoutlossofdata.Forteams,it
alsokeepsarecordofwhochangedwhichlinesofafileandpreventsdirect
overwriting of each other’s work.
SystemadministratorscankeepconfigurationfilesinCVS.Youcanmakea
change, cvscommit it,testit.Ifitfails,rollbackthechange,evenifyouonly
discover the failure six months down the track.
AdministratorscankeepaCVStreeoftheconfigurationsforserverfarms.
Addinganewserver?Justuse cvs tocheckouttheconfigtreeforthattype
ofserver.Committingallchangesalsohelpsyoukeeptrackofwhodid
what, when.
Inthissection,we’lltakealookatwhatyouneedtogetCVSupandrun-
ning quickly, both as a client and as a server.Getting Started with CVS#26
Revision Control | 53
HACK
Creating a Repository
Therepositoryneedstobehostedonamachinewithsufficientdiskspaceto
storeallyourfilesandallthedataforthechangesyouexpect.Asaruleof
thumb,puttherepositoryonapartitionwithenoughroomforthreetimes
theexpectedfinalsizeofthemodule.Thenusethe“Scottyprinciple”and
doubleyourestimate—theprojectwillexpand.Ifyouintendtostorebinary
files,multiplybyten.Afteryourfirstproject,you’llhaveafeelforhow
much space to allow.
First,ensurethatallCVSusershavevaliduseraccountsandcanaccessthe
repository machine from all the machines they intend to use.
Nowcreateyourrepositoryrootdirectory.Repositoriesareoftenstoredin
/home/cvsrootor /usr/local/cvsroot.Use cvsinit tosetupthedirectoryasa
CVS repository.
cvs -d /home/cvsroot init
DebianLinuxhasascript, cvs-makerepos,thatwillbuildarepositorybased
onpre-existingDebianconfigurationscripts.Seemancvs-makereposfor
moreinformationandmancvsconfigforanautomatedsystemforconfigur-
ing a Debian CVS repository.
Ingeneral,mostCVSserverstendtouseasinglerepository,withmulti-
pleModulescontainedwithinit.Forexample,yourrepositorymightlive
in /home/cvsroot,butcouldcontainmanyprojects(suchas tools, email,
dns, myproj, etc.).
Importing a New Module
BeforeloadingyourprojectintoCVS,consideritsstructure.Movingor
renamingfilesanddirectoriescandamagetheCVSrecordofthefiles’his-
tory.Deletingadirectorycancostyoutherecordofallitsfilesandsubdirec-
tories.Forthisreason,CVShasnofacilityformovingorrenamingfilesand
directories, or removing directories.
Makeyourinitialdirectorystructure—evenifit’sjustonedirectory.Addany
initialfilesyouwant.Fromwithintherootdirectoryofyourproject,usethe
command cvs -d repository import nameofmodule vendortag releasetag.
Formostcases,youwillnotneedtoknowaboutvendortagsandrelease
tags.CVSrequiresthemtobepresent,butyoucansimplyusethenameof
the module as the vendor tag, and the current version as the release tag.
/home/jenn$ cd example
/home/jenn/example$ cvs -d /home/cvsroot import example example_project ver_
0-1
Download from Wow! eBook <www.wowebook.com>54 | Revision Control
#27 CVS: Checking Out a Module
HACK
Environment Variables
Ifyouareonlyworkingwithonerepository,andwouldliketosaveyourself
sometyping,setthe$CVSROOTenvironmentvariabletotherepository’s
full path, like this:
export CVSROOT=/home/cvsroot/
With$CVSROOTset,youcannowomitthe -drepository portionofany
CVScommand.Ifyouneedtotemporarilyworkwithanotherrepository,
eitherresetyour$CVSROOT,orusethe -d switchtooverrideyourexisting
$CVSROOT setting.
IftheCVSrepositoryresidesonadifferentmachine,thenyouneedtotell
CVShowtoaccessit.Forremoterepositories,$CVSROOTissetto :method:
user@host:path.Itshouldlooksomethinglike :ext:jenn@cvs.example.com.
au:/home/cvs.
Bydefault,CVSuses rsh toaccessarepositoryonaremotemachine.While
thismayhavebeenaverylogicaltransportchoiceafewyearsago,running
rsh onaproductionserverisnowgenerallyconsideredaVeryBadIdea.For-
tunately,the ext accessmethodtellsCVStolookinthe$CVS_RSHenviron-
mentvariable,andrunthatprogramtoconnecttothemachinethatcontains
therepository.Mostpeoplesetthe$CVS_RSHvariableto ssh,andsetup ssh
clientkeys(asin“QuickLoginswithsshClientKeys” [Hack#66])onthereposi-
torymachine.Thisgreatlysimplifiesworkingwithyourrepository,asyou
won’t have to type in a password every time you make an update.
Incidentally,theothercommonaccessmethodis pserver,andthesedaysis
generallyusedforaccessinganonymouspublicCVSrepositories.Foran
exampleofhowtoworkwithanonymousCVS(andsetupyourownpublic
repository), see“Quick Logins with ssh Client Keys” [Hack #66].
See Also:
• CVS in a Nutshell (O’Reilly)
• Anonymous CVS
• Quick logins with ssh keys
HACK
#27
CVS: Checking Out a Module Hack #27
How to get a working copy of a CVS Module
CVSstoresthefilesinacentralrepository,butusersworkfromaworking
copy of a file.CVS: Updating Your Working Copy#28
Revision Control | 55
HACK
Makeadirectorytodoyourworkin(Itendtouse ~/cvs),then cd intothat
directory.Thecheckoutsyntaxis cvscheckoutmodule Tocheckoutamod-
ule called example, try cvs checkout example.
Thecheckoutwillputacopyofthatmodule’sfilesandsubdirectoriesinto
your cvs directory.
cvs$ ls
example
cvs$ cd example; ls
CVS src
cvs/example$ cd CVS; ls
Entries Repository Root
The cvs directoryisaspecialdirectorythatCVSusesforitsownpurposes.
CVS/EntrieslistsfilesandsubdirectoriesCVSknowsabout.CVS/Reposi-
torycontainsthepathtothecorrespondingdirectoryintherepository.
CVS/Rootcontainsthepathtotherepository,soyouwon’tneedtouse
the -d repository-path option again for these files.
NotethatCVS/Rootoverridesthe$CVSROOTenvironmentvariable,soif
youchangetherepository,youshouldcheckoutthemoduleagain.Alter-
nately,ifyou’reinthemiddleofabigeditwhenyourealizethatyourreposi-
toryneedstobechanged,trythisperlone-liner,asshownin“GlobalSearch
and Replace with Perl” [Hack #73]:
cvs/src$ ls
CVS Makefile sample.h sample.c
The src directorycontainsthesourcefilesfortheexampleproject. sample.c,
sample.h,and Makefile areordinaryfilesintheworkingcopy.Inthereposi-
tory, they are stored in a format that tracks the changes.
HACK
#28
CVS: Updating Your Working Copy Hack #28
Receiving recent changes in your CVS Module
Everydaybeforeyoustartwork,andanytimesomeoneelsemayhavemade
andcommittedchanges, cd intoyourworkingdirectoryandrun cvsupdate.
Thischecksyourworkingcopiesagainsttherepositoryfilesandimportsany
changed files for you. cvs update -d also gives you any new directories.
Update reports on the status of each file as it checks it:
U File updated successfully
A File added but not yet committed (need to run a cvs commit)
R File removed but not yet committed (need to run a cvs commit)
M Filemodifiedinyourworkingdirectory;thefileintherepositorywas
changedandyourworkingdirectoryfilewasolderthanthelasttime56 | Revision Control
#29 CVS: Using Tags
HACK
CVScheckeditortherepositoryhadchangesthatthesystemcould
safely merge
C Conflictbetweentherepositorycopy,andyourcopy,whichrequires
human intervention
? Filethefileisinyourworkingdirectorybutnottherepository;CVS
doesn’t know what to do with it
HACK
#29
CVS: Using Tags Hack #29
Tagging a Module revision in CVS
Tagsassignasymbolicnametoarevisionofafileorasetoffiles. cvstag
tagstherepositoryversionsofallthefilesinthecurrentworkingdirectory
and its subdirectories.
The cvstag commandisbasedonatimestamp.Itassignsthesymbolicname
totheversionofthefileordirectorythatisclosesttoandolderthanthe
timestamp; it does not look at the working directory.
NotethatCVSdoesn’tallowthe‘.’characterintags.Specifythefilename
after the tag name, as in the following example:
cvs tag tagname filename
cvs tag tagname
Otherwise,thecommandtagstherepositoryversionsofallfilesinthecur-
rent working directory:
cvs tag -c tagname
Usethe -c optiontocausetheprocesstoabortiftherepositorycopydiffers
from the working copy:
cvs/example$ cvs tag release-1-0 src/sample.c
cvs/example/src$ cvs tag release-1-0
cvs/example/src$ cvs tag -c release-1-0
Toretrieveataggedversion,usethe -r flagtocheckoutorupdate.Ifyou
checkoutorupdateinyourusualworkingdirectory,thetaggedversionwill
overwrite the existing files, as in:
cvs checkout -r tagname
cvs update -r tagname
Here’s an example:
cvs$ mkdir example-rel-1.0
cvs/example-rel-1.0$ cvs checkout -r release-1-0
or:
cvs/example$ cvs update -r release-1-0CVS: Making Changes to a Module#30
Revision Control | 57
HACK
HACK
#30
CVS: Making Changes to a Module Hack #30
Checking in your changes in CVS
Onceyourfilesarecheckedout,editthemandcompilethemnormally.
Applytheupdatestotherepositorywith cvscommit.Thiscommandneeds
toberunhigherinthehierarchythanallthefilesyouhavechanged—you
can always run it from the base of your working copy.
Youcanalso cvscommitfilename,whichwillcommitasinglefileorrecur-
sively commit a directory.
Differentprojectteamshavedifferentopinionsonhowoftentodoa cvs
commit.Goodrulesofthumbinclude“everytimeyouhaveacleancom-
pile,” and “every day before lunch and before you leave.”
cvs/example$ cvs commit
cvs commit: Examining .
cvs commit: Examining src
jenn@cvs.sample.com.au's password:
CVSexamineseachdirectoryandsubdirectorybelowthecurrentworking
directory.AnyfilethatCVSknowswillbecheckedforchanges.Ifyourcvs
repositoryisnotonthelocalmachine,CVSwillaskforapasswordforthe
remotemachine,unlessyouhavealreadysetupyoursshhostkeys(as
shownin“QuickLoginswithsshClientKeys” [Hack#66])andhaveelimi-
nated the need for passwords.
CVSthenopenswhichevereditoristhedefaultinyourenvironment—based
onthe$CVSEDITORor$EDITORenvironmentvariables.Addchange-
notes for the appropriate files, as in:
CVS:------------------------------------------------------------------
CVS: Enter Log. Lines beginning with 'CVS:' are removed automatically
CVS:
CVS: Committing in .
CVS:
CVS: Modified Files:
CVS: example/src/sample.h example/src/sample.c
CVS:------------------------------------------------------------------
Istronglyrecommendmeaningfulchange-notes—ifyou’retryingtodoa
rollbackandallyouhavearemessagesthatsay“fixedafewbugs,”you’ll
not know which version to roll back to without using cvs diff.
Ifthereisapotentialconflict, cvscommit fails.Correctthisbyrunninga cvs
update ontherepository—CVSwillattempttomergethefiles,andwillask
for human help if it cannot do this without losing data:
cvs server: Up-to-date check failed for 'cvs_intro.html'
cvs [server aborted]: correct above errors first!
cvs commit: saving log message in /tmp/cvst7onmJ58 | Revision Control
#31 CVS: Merging Files
HACK
HACK
#31
CVS: Merging Files Hack #31
Resolving update conflicts in CVS
IfCVScan’tmergeamodifiedfilesuccessfullywiththecopyinthereposi-
tory,itannouncestheconflictintheoutputof cvsupdate.Theoriginalfileis
storedin .#file.version inthefile’sworkingdirectory,andtheresultsofthe
merge are stored as the original filename:
cvs/example$ cvs update
jenn@cvs.example.com.au's password:
cvs server: Updating .
RCS file: /home/cvs/example/sample.c,v
retrieving revision 1.3
retrieving revision 1.4
Merging differences between 1.3 and 1.4 into sample.c
rcsmerge: warning: conflicts during merge
cvs server: conflicts found in sample.c
C sample.c
CVSwritesthemergewiththeconflictinglinessurroundedbyCVStags.
CVScan’tautomaticallymergeconflictswherethesamelineischangedin
both versions of a file:
<<<<<<< sample.c
Deliberately creating a conflict.
=======
Let's make a conflict.
>>>>>>> 1.4
HACK
#32
CVS: Adding and Removing Files and
Directories Hack #32
Adding and removing files in your Module
FilesthatCVSdoesn’tknowwhattodowitharereportedwithaquestion
markafterthecommitprocessandduringa cvsupdate.Theyneedtobe
added to the repository before CVS will recognize them.
Use cvsaddfilename tomarkanewfileforinclusion.CVSdoesn’tputthe
file in the repository until you do a cvs commit.
Directoriesareaddedwiththesamecommand.Fileswithinadirectorycan’t
be added until the directory is added.
Removing Files
Tomarkafileforremovalfromtheworkingcopies,use cvsremovefilename.
BeforeCVSwillremoveafilefromtherepository,youhavetoactuallydelete
itfromthefilesystem.CVSdoesn’tactuallyremovethefileentirely,itputsthe
file in a special subdirectory in the repository called Attic.CVS: Branching Development#33
Revision Control | 59
HACK
Removing Directories
DirectoriescannotberemovedfromtherepositoryusingCVScommands.
Ifadirectoryisnolongerrequired,emptythedirectorywith cvsremove,
anduse cvsupdate-P and cvscheckout-P whenretrievingaworking
copy. The -P flag ensures that empty directories are not retrieved.
Ifyoumust,youcanremoveadirectorybyusing rmdir ontherepository.
Dothisonacopyoftherepositoryfirstandcheckthatyouaren’tbreaking
anything:Ifthedirectoryintherepositoryhasan Attic subdirectory,you
will lose archived copies of files formerly stored there.
Ifyouremoveadirectoryfromtherepository,youshouldhaveallyourusers
removetheirexistingworkingcopies,andcheckoutfreshcopiesofthe
module.
HACK
#33
CVS: Branching Development Hack #33
Setting up a development branch in CVS
Ifyouneedtofixabuginanolderversionofyourcodewithoutchanging
currentcode,ormodifyaconfigurationsetforstagingserverswithoutmodi-
fyingthesetforyourproductionservers,youmightneedtobranchyour
modules.Abranchallowsstorageandretrievalofavariationofthemain
module,withoutaffectingthemainmodule.Changesonthebranchcanbe
merged in later. To make a branch, use cvs tag -bbranchtag, as in:
cvs/example$ cvs tag -b release-1-0-patches
Retrieveabranchusingeithercheckoutorupdate.Checkoutwillcreatea
newdirectoryforthebranch,andupdatewilloverwriteyourcurrentwork-
ing directory with the branch.
cvs checkout -r branchtag
cvs update -r branchtag
Example:
cvs/example-rel-1.0$ cvs checkout -r release-1-0-patches
cvs/example$ cvs update -r release-1-0-patches
Branchescanbemergedbackintothemaintrunk,usingtheconflictresolu-
tion system invoked by cvs update and cvs commit.
cvs checkout module
cvs update -j branchtag
Example:
/tmp/example$ cvs checkout example
/tmp/example$ cvs update -j release-1-0-patches60 | Revision Control
#34 CVS: Watching and Locking Files
HACK
Or in a single command:
cvs checkout -j branchtag module
The following example:
/tmp/example$ cvs checkout -j release-1-0-patches example
resolves any conflicts the system reports, then use cvs commit.
HACK
#34
CVS: Watching and Locking Files Hack #34
Setting up an email watch on files in CVS
Unlikemanyversioningsystems,CVSdoesn’thavefilelocking—itdoesn’t
preventsimultaneouseditingoffiles.However,youcansetfilestobe
watched,soCVSwillmailwatcherswhenafileisbeingedited.Iffilesare
beingwatched,developersneedtouse cvsedit and cvsunedit toreleaseafile
forediting.UnwatchedfilescanbeeditedwithoutnotifyingCVSinany
way.
To set up files for being watched, use:
cvs watch on (files)
cvs watch off (files)
To set yourself as a watcher, use:
cvs watch add (files)
cvs watch remove (files)
or:
cvs watch add -a edit|unedit|commit|all (files)
cvs watch remove -a edit|unedit|commit|all (files)
ThespecialCVSfilenotifydetermineswhatoccurswhenawatchedfileis
changed.Itdefaultstosendingmailtotheuser’susernameontheCVS
server.Ifyourusershaveotheraddresses,setupthefile users inthereposi-
tory’s CVSROOT directory.Entriesshouldbeintheformat user:email,one
to a line.
jenn:jenn@cvs.example.com.au
HACK
#35
CVS: Keeping CVS Secure Hack #35
Protecting your users and your code base in CVS
Remote Repositories
Iftherepositoryisonthelocalmachine,bothaccessandsecurityarefairly
straightforward.Youcansetthe$CVSROOTenvironmentvariabletotheroot
directory of the CVS repository, or call checkout with the -d directory option.CVS: Keeping CVS Secure#35
Revision Control | 61
HACK
Iftherepositoryisonaremotemachine,itisnecessarytotellCVSwhich
machineitison,andwhatmethodwillbeusedtoaccessthemachine.There
areseveralmethodsavailable,butforsecurityandsimplicity,IpreferSSH.
Thesyntaxfordefiningaremote$CVSROOTis :method:[[user]:
[password]@]hostname[:[port]]:/path/to/repository. For example:
:ext:jenn@cvs.example.com.au:/usr/local/cvsroot
(Notethat infocvs disagreesslightlywithwhatmycopyofCVSactually
does.Ihaveincludedthesyntaxthatworksforme—acolonbetweenthe
host and the path. Use the syntax that works on your system.)
TouseSSH,weusethe ext method.Thismethodusesanexternal-to-CVS rsh
or rsh-compatibleprogramtocommunicatewiththeCVSserver.TotellCVS
touseSSHinsteadofrsh,settheenvironmentvariable$CVS_RSHtoSSH.
EnsurethatSSHissetupontheserverandonallclients,thatSSHkeysare
generatedandthatusershaveusernamesandpasswordsonbothmachines.If
theusernamesarethesame,theuser@partofthe CVSROOT stringisnot
necessary. If a standard SSH port is used, the port is not necessary.
cvs -d :ext:cvs.example.com.au:/usr/local/cvsroot checkout sample<code>
Permissions
Thefilesintherepositoryareallread-only.Permissionstothosefiles
shouldn’tbechanged.Tocontrolaccess,usethedirectorypermissions.Most
administratorsmakeagroupforthepeoplewhoshouldhaveaccesstothe
module, and ensure that the group has write access for the directory.
Ifusingaremoterepository,settherootdirectoryofthemodulesetgidto
ensurethatalldirectoriesbeneathitaremadewiththecorrectpermissions.
Ifusingalocalrepository,$CVSUMASKcanbesettocontrolthepermis-
sions of files and directories in the repository.
Developer Machines
Securingtheprojectinvolvessecuringtherepositoryandsecuringall
checkedoutcopies—typicallyyourdeveloper’smachines.It’snotenoughto
ensurethattherepositoryissafeandalltransmissionsareproperly
encryptedifsomeonecanwalkintoyourdeveloper’sofficeonhisdayoff
andburnaCDofyourcode.MaintaintheusualphysicalandNet-based
securityforyourdevelopmentmachines,prototypeanddemonstrationcop-
ies, and any other places the code gets checked out to.62 | Revision Control
#36 CVS: Anonymous Repositories
HACK
HACK
#36
CVS: Anonymous Repositories Hack #36
Create your own read-only anonymous CVS repository
Creating an Anonymous Repository
The pserver accessmethodallowsuserstologintoremoterepositoriesby
supplyingausernameandpasswordthatischeckedagainstapasswordfile
thatCVSmaintains,oragainstthesystem’s /etc/passwd.The pserver method
unfortunatelypassesallcredentialsintheclear,soatbestanintervening
snoopermightcapturealogintoyourCVSrepository,oratworstcould
compromisetheentiremachine.Forthisreason,mostpeopleusesshas
theirtransportwhenusingremoterepositories.Pleasesee“CVS:Keeping
CVS Secure” [Hack #35], for further details.
Obviously,ifyouwanttoprovideread-onlyaccesstoyoursourcetreetothe
public,using ssh asyourtransportwouldbeunnecessary(andimpractical).
Thisiswhere pserver showsitsrealusefulness:allowingeasyanonymous
repository access.
BeforewegetanonymousCVSrunning,firstwe’llneedtosettherepository
machine up to use the traditional pserver method.
Installing pserver
Aswe’llbeusing pserver foranonymousCVSaccess,we’llneedtocreatea
userthathasnopermissionstowritetoanythinginyourrepository.Create
ausercalledanonymous(orif9-letterusernamesbotheryou,cvsanonis
anothercommonchoice.)Setitsshellto /bin/true,itshomedirectoryto
somethinginnocuous(like /var/empty),putitinitsowngroup,andlockits
password(a passwd-l isaquickwaytodothat.)Thisuserwillneverlogin;
it’s just a placeholder account for CVS to setuid to later.
Nextwe’llcreateapasswordfileforCVS.Putthefollowinginafilecalled
CVSROOT/passwd, under your repository directory:
anonymous:23MLN3ne5kvBM
If you created a user account called cvsanon, use this line instead:
anonymous:23MLN3ne5kvBM:cvsanon
IntheCVS passwd file,thelefthandentryistheCVSloginname,followed
bytheencryptedpassword,andfinallyendinginanoptionalsystemlogin
nametomaptheCVSloginto.Ifitisomitted,CVSwilllookupthefirst
entryinthesystem’spasswordfileandusethat.Theencryptedstringisthe
word anonymous.CVS: Anonymous Repositories#36
Revision Control | 63
HACK
Tobeabsolutelysurethattheanonymoususercan’tevermakechangesto
therepository,addtheline anonymous tothe CVSROOT/readers fileunder
yourrepository.Thisfileflagsanyuserscontainedwithinit(oneperline)as
read-only users.
NowwewanttotellCVStoneveracceptregularsystemusersunderthe
pserver method(topreventwaywardusersfromhabituallyusingtheirsys-
temloginswithpserver.)Thisissetupin CVSROOT/config,underyour
repositorydirectory.Uncommentthelinethatsays SystemAuth=no andthen
onlyusersspecifiedin CVSROOT/passwd canloginusingpserver.Notethat
thiswillhavenoeffectonCVSusersthatuse ext and ssh;theystillusesim-
plefilesystempermissionsforaccesscontrol,andneverconsulttheCVS
passwd file.
Finally,wecantellthesystemtoactuallyaccept pserver connections.CVS
doesn’teverrunasadaemon;itexpectstobelaunchedfrom inetd.Itruns
on port 2401, so add the following line to your /etc/services:
pserver 2401/tcp
And add this entry to /etc/inetd.conf:
pserver stream tcp nowait root /usr/bin/cvs cvs --allow-root=/usr/local/
cvsroot pserver
Substituteyourrepository’sdirectoryfor /usr/local/cvsroot.Nowjust skill
-HUPinetd (andsee“GettingStartedwithRCS” [Hack#23] ifyoudon’thave
skill installed) and away you go.
Using a Remote pserver
Totestyournewanonymousrepository,firstsetyour$CVSROOTenviron-
ment variable to this:
:pserver:anonymous@your.machine.here:/usr/local/cvsroot
Beforeyoucandoa cvscheckout,you’llfirstneedtologintopserverwith
cvslogin.Whenpromptedforapassword,enter anonymous.Nowproceed
with cvscheckoutmodule,andyoushouldseeanupdateasnormal.Verify
thatyoucan’tperforma cvscheckin,andyou’rereadytopublishyouranon-
ymous CVS details to the world.64
Chapter3 CHAPTER THREE
Backups
Hacks #37–44
Beassuredthatoneday,theunthinkable will happen.Harddriveswear.
Tapesstretchandbreak.Youmightevenaccidentallypassthewrongswitch
toacommand(orbeinthewrongdirectory,oronthewrongmachine)and
suddenlyfindyourselfcursingathavinghittheEnterkey,asyourdataevap-
oratesintotheether.Itisatthesemomentsthatyoulearnthevalueofa
properly designed and executed backup solution.
Whilewecangetyoustartedwithsomeinterestingapproachestobacking
upyourdata,thissectionisbynomeanscomprehensive.Asasystems
administrator,youshoulddesignabackupplanthatfitstheneedsofyour
organization,implementtheplan,andcontinuallyreviseitasnecessaryas
timegoeson.Definitelydonotassumethatjustbecauseyousetupa cron
jobtorunabackupthatthebackupsthemselvesaregood(oreventhatthe
jobitselfisrunningproperly).Watchyourlogs.Buynewbackupmedia
beforetheoldmediadevelopsproblems.Makeitapolicytotestyourback-
upsbydoingafullrestorefrombackupasoftenasyoudeemnecessaryto
sleepwellatnight.Andeventhen,considerkeepinganothercopyofyour
data,off-siteifneedbe.Foranextremelydetailedlookateverythingthat
goesintobuildingasecuredatabackuppolicy,besuretocheckout Unix
Backups & Recovery by W. Curtis Preston (O’Reilly).
Inthissection,we’lltakealookatsomemethodsforkeepingcopiesofyour
datareadyforwhenthatdayofreckoningcomes.Everyinstallationisdiffer-
ent,andeachsiteneedsitsowndegreeofbackupfunctionality(ranging
fromweeklydifferentialsto5-minutesnapshots).Whilewewon’ttryto
definetheperfectbackuppolicyforyoursite,wewillseesometoolsyoucan
use to develop your own.Backing Up with tar over ssh#37
Backups | 65
HACK
HACK
#37
Backing Up with tar over ssh Hack #37
Copy arbitrary bits of the filesystem between servers using ssh and tar
Shuffling files between servers is simple with scp:
root@inky:~# scp some-archive.tgz blinky:/
Or even copying many files at once:
root@pinky:~/tmp# scp clyde:/usr/local/etc/* .
But scp isn’tdesignedtotraversesubdirectoriesandpreserveownershipand
permissions.Fortunately, tar isoneoftheveryearly(andIMHO,mostbril-
liant)designdecisionsin ssh tomakeitbehaveexactlyasanyotherstan-
dardUnixcommand.Whenitisusedtoexecutecommandswithoutan
interactiveloginsession, ssh simplyacceptsdataonSTDINandprintsthe
resultstoSTDOUT.Thinkofanypipelineinvolving ssh asaneasyportalto
themachineyou’reconnectingto.Forexample,supposeyouwantto
backup all of the home directories on one server to an archive on another:
root@inky~# tar zcvf - /home | ssh pinky "cat > inky-homes.tgz"
Orevenwriteacompressedarchivedirectlytoatapedriveontheremote
machine:
root@blinky~# tar zcvf - /var/named/data | ssh clyde "cat > /dev/tape"
Supposeyouwantedtojustmakeacopyofadirectorystructurefromone
machinedirectlyintothefilesystemofanother.Inthisexample,wehavea
workingApacheonthelocalmachinebutabrokencopyontheremoteside.
Let’s get the two in sync:
root@clyde:~# cd /usr/local
root@clyde:/usr/local# tar zcf - apache/ \
| ssh pacman "cd /usr/local; mv apache apache.bak; tar zpxvf -"
Thismoves /usr/local/apache/ on pacman to /usr/local/apache.bak/,thencre-
atesanexactcopyof /usr/local/apache/ fromclyde,preservingpermissionsand
theentiredirectorystructure.Youcanexperimentwithusingcompressionon
bothendsornot(withthe z flagto tar),asperformancewilldependonthe
processingspeedofbothmachines,thespeed(andutilization)ofthenet-
work, and whether you’re already using compression in ssh.
Finally,let’sassumethatyouhavealargearchiveonthelocalmachineand
wanttorestoreittotheremotesidewithouthavingtocopyittherefirst
(supposeit’sreallyhuge,andyouhaveenoughspacefortheextractedcopy,
but not enough for a copy of the archive as well):
root@blinky~# ssh pinky "cd /usr/local/pacland; tar zpvxf -" \
< really-big-archive.tgz66 | Backups
#38 Using rsync over ssh
HACK
Or alternately, from the other direction:
root@pinky:/usr/local/pacland# ssh blinky "cat really-big-archive.tgz" \
| tar zpvxf -
Ifyouencounterproblemswitharchivescreatedorextractedontheremote
end,checktomakesurenothingiswrittentotheterminalinyour ~/.bashrc
ontheremotemachine.Ifyouliketorun /usr/games/fortune orsomeother
programthatwritestoyourterminal,it’sabetterideatokeepitin ~/.bash_
profile or ~/.bash_login thanin ~/.bashrc,becauseyou’reonlyinterestedin
seeingwhatfortunehastosaywhenthereisanactualhumanbeinglogging
inanddefinitelynotwhenremotecommandsareexecutedaspartofapipe-
line.Youcanstillsetenvironmentvariablesorrunanyothercommandyou
likein ~/.bashrc,aslongasthosecommandsareguaranteednevertoprint
anything to STDOUT or STDERR.
Using ssh keystoeliminatetheneedforpasswordsmakesslingingaround
arbitrarychunksofthefilesystemeveneasier(andeasilyscriptableincron,if
you’resoinclined).Seethehackslistedbelowforanexampleofhowtomake
it very easy (and secure) to connect to any of your servers with ssh.
See also:
•“Quick Logins with ssh Client Keys” [Hack #66]
•“Turbo-mode ssh Logins” (#67)
•“Using ssh-Agent Effectively” [Hack #68]
HACK
#38
Using rsync over ssh Hack #38
Keep large directory structures in sync quickly with rsync
While tar over ssh isidealformakingremotecopiesofpartsofafilesys-
tem, rsync isevenbettersuitedforkeepingthefilesysteminsyncbetween
twomachines.Typically, tar isusedfortheinitialcopy,and rsync isused
topickupwhateverhaschangedsincethelastcopy.Thisisbecause tar
tendstobefasterthan rsync whennoneofthedestinationfilesexist,but
rsync ismuchfasterthan tar whenthereareonlyafewdifferencesbetween
the two filesystems.
To run an rsync over ssh, pass it the -e switch, like this:
root@rover:~# rsync -ave ssh greendome:/home/ftp/pub/ /home/ftp/pub/
Noticethetrailing / onthefilespecfromthesourceside(ongreendome.)
Onthesourcespecification,atrailing / tells rsync tocopythe contents ofthe
directory,butnotthedirectoryitself.Toincludethedirectoryasthetop
level of whatever is being copied, leave off the /:
root@village:~# rsync -ave ssh bcnu:/home/six .Archiving with Pax#39
Backups | 67
HACK
Thiswillkeepacopyofthe ~root/six/ directoryonvillageinsyncwithwhat-
ever is present on bcnu:/home/six/.
Bydefault, rsync willonlycopyfilesanddirectories,butnotremovethem
fromthedestinationcopywhentheyareremovedfromthesource.Tokeep
the copies exact, include the --delete flag:
six@jammer:~/public_html# rsync -ave ssh --delete greendome:~one/reports .
Nowwhenoldreportsareremovedfrom ~one/reports/ ongreendome,
they’realsoremovedfrom ~six/public_html/reports/ onjammer,everytime
thiscommandisrun.Ifyourunacommandlikethisin cron,leaveoffthe v
switch.Thiswillkeeptheoutputquiet(unless rsync hasaproblemrunning,
in which case you’ll receive an email with the error output).
Using ssh asyourtransportfor rsync traffichastheadvantageofencrypting
thedataoverthenetworkandalsotakesadvantageofanytrustrelation-
shipsyoualreadyhaveestablishedusing ssh clientkeys.Forkeepinglarge,
complexdirectorystructuresinsyncbetweentwomachines(especially
whenthereareonlyafewdifferencesbetweenthem), rsync isaveryhandy
(and fast) tool to have at your disposal.
See also:
•man rsync
•“Quick Logins with ssh Client Keys” [Hack #66]
•“Using ssh-Agent Effectively” [Hack #68]
•“Automated Snapshot-Style Incremental Backups with rsync” [Hack #42]
HACK
#39
Archiving with Pax Hack #39
Make easy, portable archives using pax
pax standsfor“portablearchiveexchange,”asitwasdesignedspecificallyto
allowportabilitybetweendifferentversionsofUnix.There’salsoabitof
wryhumorinthename,as pax attemptstobringsome“peace”tothelong-
standingbattleoverwhichisbetter: tar or cpio.The pax utilitycanbeused
tocreateeithertypeofarchive,andduringarestore,itautomaticallydetects
thetypeofarchiveforyou.We’llstartwithsomeexamplesofbasic pax
usage, then move on to some fancier stuff.
Creating Archives
Tobackupthecontentsofyourhomedirectory,invokewritemodeusing
the w switch:
cd
pax -wf home.pax .
Download from Wow! eBook <www.wowebook.com>68 | Backups
#39 Archiving with Pax
HACK
Inthisexample,Iwenttomyhomedirectorywith cd,thentold pax towrite
(w)toafile(f)named home.pax thecontentsofthecurrentdirectory(.).
Whenyouuse pax,it’sveryimportanttoremembertoincludethat f switch
toindicatethenameofthearchiveyou’dliketocreate.Ifyouforgetthef,
weirdcharacterswillbesenttoyourscreen,accompaniedbyhorrible,
painednoises.Also,ifyouwanttowatchas pax doesitsthing,simplyadd
the v, or verbose, switch to the switch portion of the command.
To see what type of file you’ve just created, use the file command:
file home.pax
home.pax: POSIX tar archive
Toseethecontentsofthatarchive,tell pax whicharchivefileyou’dliketo
view, using the f switch:
pax -f home.pax |more
Sincemyarchivefileisratherlarge,Ipipedthisoutputtothe more command
soIcouldreadthecontentsofthearchiveonepageatatime.Ifyoualso
includethe v switch,you’llget ls-l–typeoutputofthearchivecontents.Again,
don’tforgettospecifythenameofthearchivewiththe f switch,ornothing
will happen, except that you’ll lose your prompt until you press Ctrl-c.
Expanding Archives
Torestore(orusereadmodeon)anarchive,first cd intothe destination
directory,thenusethe r switch.Forexample,I’llrestorethebackupnamed
home.pax into the test subdirectory of my home directory:
cd test
pax -rvf ~/home.pax
The pax utilitycanalsorestore tar and cpio archives.Itisabletoautomati-
callydetectthecorrectformatforyou;however,youshouldusethe file util-
itybeforeattemptingtherestoretodeterminewhetherornotthearchiveis
compressed. If it is, you’ll need to include the z switch.
Asanexample,Ihaveafilecalled backup.old locatedinmyhomedirectory
(~). I’ll first use the file utility:
file backup.old
backup: gzip compressed data, deflated, last modified:
Sat Aug 17 14:21:12 2002, os: Unix
Sincethisbackupiscompressed,I’llusethiscommandtorestoreittothe
test directory:
cd test
pax -rvzf ~/backup.oldArchiving with Pax#39
Backups | 69
HACK
I have another file in my home directory called backup:
file ~/backup
backup: cpio archive
This file isn’t compressed, so i’ll restore it, like so:
pax -rvf ~/backup
Thefactthatthefirstbackuphappenedtobea tar archiveandtheseconda
cpio archivedidn’tconfuse pax;however,Iwouldhavereceivedsome
strangeerrormessagesifIhadforgottentoinform pax thatthefirstarchive
was compressed.
Interactive Restores
Youcandosomeprettyfunkythingswhenrestoringwith pax.Forexam-
ple,youcandoaninteractiverename/restorebyincludingthe i switch.Issu-
ing the following command:
pax -rif ~/backup
willstartaninteractiverestoreofthearchivenamed backup intothecurrent
directory.Ininteractivemode, pax willdisplaythenameofeachfile,oneat
atime,andpromptyoutoeitherrenameitasit’srestored,restoreitwiththe
original name, or to skip it and not restore it:
ATTENTION: pax interactive file rename operation.
drwxr-xr-x Aug 17 15:08 .
Input new name, or a "." to keep the old name, or a "return" to skip this
file.
Input >
Skipping file.
Here,IpressedEnterasIdidn’twanttochangethenameof“.”orthecur-
rent directory:
ATTENTION: pax interactive file rename operation.
drwxr-xr-x Jul 26 16:10 file1
input new name, or a "." to keep the old name, or a "return" to skip this
file.
input > old
Processing continues, name changed to: old
ATTENTION: pax interactive file rename operation.
-rw-r--r-- Jun 11 00:20 file2
input new name, or a "." to keep the old name, or a "return" to skip this
file.
input > .
Processing continues, name unchanged.
You’llnotethatIchangedthenameof file1 to old andkept file2 asis.Alist-
ingoftherestoreddirectorywillshowtwofiles:onenamed old andone
named file2.70 | Backups
#39 Archiving with Pax
HACK
Recursively Copy a Directory
Oneofthemostpowerfulfeaturesof pax isthatitisabletoveryquickly
copyacompletedirectorystructuretoanotherportionofyourharddrive,
using copy mode. To use copy mode:
1. cd into the source directory
2.Ensure the destination directory exists; if it doesn’t, use mkdir to create it
3.Issue this command: pax -rw . destination_directory
Notethatyoudon’tincludethe f switchincopymode,asanarchivefile
doesn’tgetcreated.Instead,theolddirectorystructureisdirectlyrecreated
intothenewdirectorystructure.Thiscanbemucheasiertorememberthan
the tar equivalent:
tar cf - . | (cd destination_directory; tar vpxf -)
Also note that you never want to do this:
cd
mkdir test
pax -rw . test
Inthepreviousexample,I cd’dintomyhomedirectory,madeasubdirec-
torynamed test,theninvokedcopymode.Indoingso,Iendedupinanend-
lessloopof test subdirectories,eachcontainingthecontentsofmyhome
directory.IfIhadn’tinterruptedthiscyclewithaCtrl-c, pax wouldhave
continuedadinfinitum,whereinfinitumisdefinedasthepointwhereIrun
out of disk space. That’s what this section of man pax refers to.
Warning:the destination directorymustnotbeoneofthefileoperandsora
memberofafilehierarchyrootedatoneofthefileoperands.Theresultofa
copy under these conditions is unpredictable.
However, this works beautifully and almost instantaneously:
su
Password:
cd ~user1/big_project
mkdir ~user2/big_project
chown user2 ~user2/big_project
pax -rw . ~user2/big_project
Voila,theentire big_project directorystructureisnowalsointhesecond
user’shomedirectory.Whenusingcopymode,you’llhavetobecomethe
superuserasyou’llbecopyingoutofyourhomedirectory,soyoucanavoid
theendlessloopsituation.Ifyouhavetomakethenewdirectory,itwillbe
ownedbyroot;ifneedbe,usethe chown commandlikeIdidtoensurethat
ithasthedesiredownershipbeforedoingthecopyoperation.You’llalso
wanttotakealookatmanpaxfirsttoseehowyouwanttohandletheper-
missions of the copied directory structure.Archiving with Pax#39
Backups | 71
HACK
Itisalsopossibletointeractivelycopyadirectorystructurebyincludingthe i
switch:
pax -rwi . ~user2/big_project
Similarlytothepreviousinteractiveexample, pax willdisplayeachfile-
name,oneatatime,soyoucandecidewhichfilestocopyoverandwhich
files to rename as you do so.
Incremental Backups
Now,let’sdosomethingusefulwiththe pax command.I’lldemonstrate
howtocreateanincrementalbackupsystem.Inthisexample,theuser
“genisis”wouldliketobackupanychangesshemadetoherhomedirec-
tory on a daily basis.
First, I’ll become the superuser to create a directory to hold the backups:
su
Password:
mkdir /usr/backups
I’llthencreateasubdirectoryandgivetheuser“genisis”ownershipofthat
subdirectory:
mkdir /usr/backups/genisis
chown genisis /usr/backups/genisis
I’llthenleavethesuperuseraccountandastheuser“genisis,” cd intomy
home directory:
exit
cd
I’llthendoafullbackupofmyhomedirectoryandsaveittoanarchivefile
called Monday:
pax -wvf /usr/backups/genisis/Monday .
NowthatIhaveafullbackup,Icantakedailyincrementalbackupstojust
backupeachday’schanges.SowhenI’mfinishedwithmyworkonTues-
day, I’ll issue this command:
pax -wv -T 0000 -f /usr/backups/genisis/Tuesday .
NoticethatIincludedthetimeswitch(-T)andspecifiedatimeofmidnight
(0000).Thistells pax toonlybackupthefilesthathavechangedsincemid-
night,soitwillcatchallofthefilesthatchangedtoday.OnWednesday,I’ll
repeat that command but will change the archive name to Wednesday.
Ifyouhavethediskspaceandwanttokeepbackupsforlongerthanaweek,
modifyyourarchivenamestosomethinglike:Aug01,Aug02,etc.It’sstilla
goodideatodoafullbackuponceaweek,followedbyincrementalback-
upstheotherdaysofthatweek.Ifdiskspaceisanissue,includethe z switch72 | Backups
#40 Backing Up Your Boot Sector
HACK
sothebackupswillbecompressed.Alsonotethatthe T switchcanbemuch
pickier than I’ve demonstrated; see man pax for the details.
Skipping Files on Restore
To restore all of the files except file3, use this command:
pax -rvf ~/backup -c './file3'
The c switchistheexceptionswitch.Notethatyourexceptionpattern(in
mycase, file3)needstobeenclosedinsinglequotes(thekeynexttoyour
Enterkey).EitherusetheliteralpatternlikeIdid(to pax,thisfileisknown
as ./file3, not file3) or use a wildcard, like so:
pax -rvf ~/backup -c '*file3'
Ifyouuseawildcard(*)atthebeginningofyourpatternasintheabove
example,youwillexcludeallfilesthatendwith“file3”—forexample: file3,
myfile3, thatfile3.
Youcanalsospecifywhichfiletorestorebyusingthe n,orpatternmatch-
ing, switch. The following will just restore file2:
pax -rvf ~/backup -n './file2'
The n switchdiffersfromthe c switchinthatitwillonlyrestorethefirstfile
thatmatchesthepattern.Thismeansthatthiscommandwillnotrestore
file3, myfile3, and thatfile3:
pax -rvf ~/backup -n '*file3'
Since file3 isthefirstfiletomatchtheexpression,itwillbetheonlyfilethat
will be restored.
The c and n switchesarealsousefulwhencreatinganarchive;usethemto
specifywhichfileyou’dliketobackup,orwhichfile(s)youdon’twantto
back up.
See also:
•Originalarticle: http://www.onlamp.com/pub/a/bsd/2002/08/22/freeBSD_
Basics.html
• pax sourcecode(partoftheast-openpackage), http://www.research.att.
com/sw/download/
HACK
#40
Backing Up Your Boot Sector Hack #40
Keep a copy of your boot sector packed away for a rainy day
Installingabootloader(suchasLILO)canbetrickierthanonemightlike.
ParticularlywhenusingIDEhardware,it’seasytogetyourselfintotroubleBacking Up Your Boot Sector#40
Backups | 73
HACK
andworkasystemintoastatethatmakesitimpossibletobootwithouta
rescue disk.
OnecommonmistakewhenusingIDEhardwareistoinstallakernelona
partitionthatextendsbeyondthe1024thcylinder.Thesymptomisvery
strange,becauseamachinewillbootfineatfirst,butinstallingakernellater
(afterthemachinehasbeenusedforsometime)makesLILOthrowanerror
andwillrefusetocomeuponthenextboot.Thiscanbeaveryconfusing
symptom,since“itusedtowork.”Mostlikely,thekernelthatwasinstalled
whenthesystemwasbuiltthefirsttimehappenstoresideonaspaceon
diskbeforecylinder1024.Afterthesystemsoftwareanduserdataare
installed,thediskbeginstofillup.Whenthediskcontainsabout500MB
(or1GBonsomeBIOS)ofdata,anynewkernelswillnecessarilylie(atleast
inpart)beyondcylinder1024—whichisinaccessibletotheBIOSatboot
time.
ModernversionsofLILOwillrefusetoinstallsuchakernel,butsome
olderversionssimplythrowawarning,andinstallanyway.Onewayto
dealwiththisistomakeasmall(say,10MB)partitionatthebeginningof
yourinstallprocess(as /dev/hda1,thefirstpartitiononthedisk),and
mountitunder /boot.Nowwhenyouinstallnewkernels,alwayscopy
themto /boot,andtheyareguaranteedtoworkwithyourBIOS,asthey
will necessarily be contained well before the 1024 cylinder limit.
Atanyrate,installingthebootloadershouldn’tbetakenlightly.Onceyou
haveyourbootloaderupandworkingasyouwantit,youshouldconsider
making a backup of the entire boot sector of your boot drive.
For IDE:
dd if=/dev/hda of=bootsector.bin bs=512 count=1
For SCSI:
dd if=/dev/sda of=bootsector.bin bs=512 count=1
Alternately, you can copy the boot sector directly to a floppy disk:
dd if=/dev/hda of=/dev/fd0 bs=512 count=1
Beextremelycarefulwhenspecifyingyourifandofstatements,asconfusing
themcouldobliterateyourbootsector.It’sprobablyabetterideatocreateafile
andcopyittoremovablemediaasyouwouldanyotherfile.Trustingafloppy
diskwithcriticalsystemdatamaybeafunpartytrick,butafteryourfirst(or
thousandth) floppy fails at a critical moment, you’ll wish you had another copy.
Nowifyoueverendupwithamungedbootsector,youcanrestoreit
quickly (after booting from rescue media) like this:
dd if=bootsector.bin of=/dev/hda74 | Backups
#41 Keeping Parts of Filesystems in sync with rsync
HACK
Or if you backed it up to floppy directly, against medical advice:
dd if=/dev/fd0 of=/dev/hda bs=512 count=1
Naturally, substitute sda for hda if you’re using SCSI.
See also:
•Lilo’sREADME(orotherformatofdocumentationunder doc/ fromthe
lilo distribution)
HACK
#41
Keeping Parts of Filesystems in sync with
rsync Hack #41
Usersyncoversshtomirrorexactlywhatyouwant,toanynumberofservers
ForO’Reilly’swebpublishingsystem,webuiltaweb“cluster”thatspreads
theworkofonemachineacrossseveral,increasingbothperformanceand
reliability.Buthowdoyoukeeptheactualdatainthefilesystemofeach
server in sync with each other?
OnemethodistouseNFStoprovideacommonfilesystemforyourdata.
Whilemakingitsimpletoupdatecontentacrossalloftheserverssimulta-
neously,NFShassomesignificantdrawbacks.FilelockingonNFSisnotori-
ouslytricky.PerformancetuningonNFSisgenerallyconsideredsomething
ofablackart.(Whenitworks,it’sverygood,andwhenitdoesn’t...that’s
whywe’reoncall,right?)Butprobablythebiggestdrawbacktoamono-
lithicNFSserveristhatitintroducesasinglepointoffailureforyourentire
enterprise.IftheNFSserverbecomesunavailable(oroverloaded)forany
reason, all of the machines dependent on it will suffer.
Onealternativeapproachistoasynchronouslyupdateserversusingatool
suchas rsync.Withthecostofdiskspaceatanall-timelow,havingalocally
cachedcopyofyourdatamakessense:notonlywillthesinglepointoffail-
urebeeliminated,butthefileswillbeservedmuchfasterfromthelocaldisk
than they could be from network storage.
Itisverystraightforwardtoaddan rsync jobto cron oneachofthewebserv-
ers, containing something like this:
rsync -ae ssh master.machine.com:/usr/local/apache/htdocs/ \
/usr/local/apache/htdocs/
Assumingthatyouhaveyour ssh keyssetupinadvance(see“QuickLogins
withsshClientKeys” [Hack#66]),thiswillupdatethelocalApachedocument
rootwiththecurrentcopyon master.machine.com overanencrypted ssh ses-
sion.Aslongasupdatesarealwaysmadeto master.machine.com,theywill
be copied faithfully to each of your other servers on the next pass.Keeping Parts of Filesystems in sync with rsync#41
Backups | 75
HACK
Thebiggestdrawbacktothisapproachisthatupdatesmusthappenasyn-
chronously.Ifthisjobrunseveryfiveminutes,thenthecopywillbeatmost
fiveminutesold(andaboutthreeminutesold,onaverage).Aslongasyour
applicationcantoleratethe“bethereinaminute”natureof rsync’inglarge
filestructures,thismethodcanbuyyouhugewinsintermsofactualserver
performanceandreliability.Onestrongcaveattousingabare rsync in cron
isthatyoumusttakecarethatthejobfinishesbeforethenextoneisrun.
Otherwise,ifaservergetsparticularlybusy,itcanentera downwardspiral
wherelaunchinganew rsync drivesuptheload,whichmakestherunning
rsync takelonger,whichmeansthatitisstillrunningwhenthenextone
runs, which drives up the load.
Using rsync hasacoupleofadvantagesoverusing tar forthisapplication.
Using tar over ssh (see“BackingUpwithtaroverssh” [Hack#37])isgenerally
quickerwhenmakingafullcopyofadirectorytreefromscratch,but rsync
ismuchfasterwhenonlyafewfileshavechangedonthemastercopy.It
runsananalysispassfirsttodeterminewhichfileshavebeenupdatedand
then only transfers the changes.
Butsupposethatourwebclusterhasmorecomplicatedrequirements.Assume
thatweareattemptingtodividetheload(see“DistributingLoadwithApache
RewriteMap” [Hack#99])ofservingpagesintotwogroups:ApplicationServers
(whichrunaheavymod_perl-enabledApache,drivingourcontentmanage-
mentsystem)andlightweightfront-endApacheservers,whichonlyservecop-
iesofstaticdata(suchasimagesandfilearchives).Inthiscase,itwouldbea
wasteofspacetocopytheentiredocumenttreetoalloftheservers,sincethe
front-end machines will only ever serve a few particular types of files.
Usingthe exclude-from featureof rsync,youcansupplyafilethatspecifies
whatpartsofthefilesystem rsync willconsiderwhensyncing.Createafilethat
looks something like this in /usr/local/etc/balance.front for your static servers:
### Stuff we definitely don't want to mirror.
#
- logs/
### the entire document root
#
+ /usr/
+ /usr/local/
+ /usr/local/apache/
+ /usr/local/apache/htdocs/
+ /usr/local/apache/htdocs/**
### user public_html directories
#
+ /home/
+ /home/*/
+ /home/*/public_html/76 | Backups
#41 Keeping Parts of Filesystems in sync with rsync
HACK
+ /home/*/public_html/**
# Images, archives, etc.
#
+ *.jpg
+ *.gif
+ *.png
+ *.pdf
+ *.mp3
+ *.zip
+ *.tgz
+ *.gz
# Exclude everything else.
#
- *
And create a similar file for syncing to your Application Servers:
### Stuff we definitely don't want to mirror.
#
- logs/
- *.tmp
- *.swp
### the entire document root
#
+ /usr/
+ /usr/local/
+ /usr/local/apache/
+ /usr/local/apache/htdocs/
+ /usr/local/apache/htdocs/**
# Exclude everything else.
#
- *
Now,createalistofeachtypeofserver(frontandback),onemachineper
line.Callthefiles /usr/local/etc/servers.front and /usr/local/etc/servers.back,
respectively.
For example, put this in servers.front:
tiberius
caligula
and this in servers.back:
augustus
claudius
germanicus
posthumous
castor
Finally,ratherthancalling rsync directlyfromcrononeachofyourweb
servers, try this bit of shell code in cron on the authoritative master machine.Keeping Parts of Filesystems in sync with rsync#41
Backups | 77
HACK
Listing: Balance-push.sh
#!/bin/bash
#
# balance-push - Push content from the master server (localhost)
# to multiple front- and back-end servers, in parallel.
#
# $FRONT_END lists the servers that receive the front-end (e.g. static
content) updates.
#
FRONT_END=$(cat /usr/local/etc/servers.front)
#$BACK_ENDliststhehoststhatreceivethefullback-end(e.g.everything)
updates.
#
BACK_END=$(cat /usr/local/etc/servers.back)
# $TARGET specifies the filesystem root on the remote host to push to.
# Normally, you want this to be /, unless you're doing testing.
#
TARGET=/
# $EXCLUDE specifies the prefix of the per-mode rsync exclude files.
# For example, if your exclude files are /usr/local/etc/balance.front and
# /usr/local/etc/balance.back, set this to "/usr/local/etc/balance". The
# per-mode extensions will be added.
#
EXCLUDE=/usr/local/etc/balance
# $LOCK_DIR specifies a path to put the lock files in.
#
LOCK_DIR=/var/tmp
######## Ignore the shell functions behind the curtain. ########
PATH=/bin:/usr/bin:/usr/local/bin
lock () {
local lockfile="$LOCK_DIR/balance.$1.lock"
if [ -f $lockfile ]; then
if kill -0 $(cat $lockfile); then
echo "$0 appears to be already running on $1."
echo "Please check $lockfile if you think this is in error."
exit 1
else
echo "$0 appears to have completed for $1 without cleaning up its lockfile."
fi
fi
echo $$ > $lockfile
}78 | Backups
#41 Keeping Parts of Filesystems in sync with rsync
HACK
unlock () {
rm -f $LOCK_DIR/balance.$1.lock
}
push_files () {
local mode=$1 host=$2
if [ ! "$mode" -o ! -r "$EXCLUDE.$mode" ]; then
echo "$0 $$: mode unset for $host!"
return
fi
if [ ! "$host" ]; then
echo "$0 $$: host unset for push $mode!"
return
fi
lock $host
rsync --archive --rsh=ssh --delete --ignore-errors --whole-file \
--exclude-from="$EXCLUDE.$mode" / ${host}:${TARGET}
unlock $host
}
push_tier () {
local mode=$1 host_list=$2
for host in $host_list; do
$SHELL -c "push_files $mode $host" &
done
}
export -f lock unlock push_files
export TARGET EXCLUDE LOCK_DIR PATH
[ "$FRONT_END" ] && push_tier front "$FRONT_END"
[ "$BACK_END" ] && push_tier back "$BACK_END"
#
# Fin.
#
Thisscript(callit balance-push)willmanageyour rsync jobsforyou,ensur-
ingthatserversdon’t“lap”themselvesoneachrun.Itwillpushtheproper
filestoeachgroup,dependingonwhateveryouspecifyinthefilesin /usr/
local/etc/.Ifitfindsaserverthathasn’tfinisheditslastrun,itwillcontinue
toskiptheserveruntilithasfinished(andissueyouanemailtothateffect
via cron’s MAILTO feature).
Theloadonyourmasterwilllikelygoup,dependingonhowmanymachines
you’resyncingto(aseachserverrequiresbothan rsync andan ssh session.)Automated Snapshot-Style Incremental Backups with rsync#42
Backups | 79
HACK
Butpracticallyspeaking,onaproductionnetworkservingmillionsofhitsa
day, the load introduced to each individual web server is negligible.
HACK
#42
Automated Snapshot-Style Incremental
Backups with rsync Hack #42
Use rsync to create fast, small, and safe snapshots of your filesystem
Hereisamethodforgeneratingautomaticrotatingsnapshot-stylebackups
onaLinuxserver.Snapshotbackupsareafeatureofsomehigh-endindus-
trialstrengthfileservers;theycreatethe illusion ofmultiplefull(uptoown-
ership/permission)backupsperdaywithoutthespaceorprocessing
overhead.Allofthesnapshotsareread-onlyandareaccessibledirectlyby
users as special system directories.
Sincemakingafullcopyofalargefilesystemcanbeatime-consumingand
expensiveprocess,itiscommontomakefullbackupsonlyonceaweekor
onceamonth,andstoreonlychangesontheotherdays.Thistechniqueis
calledmaking“incremental”backups,andissupportedbythevenerableold
dump and tar utilities, along with many others.
ThestandardGNUfileutils cp commandcomeswitha -l flagthatcausesitto
create(hard)linksinsteadofcopies(itdoesn’thard-linkdirectories,though,
whichisgood;youmightwanttothinkaboutwhythatis).Anotherhandy
switchforthe cp commandis -a (archive),whichcausesittorecursethrough
directories and preserve file owners, timestamps, and access permissions.
Together,thecombination cp-al makeswhatappearstobeafullcopyofa
directorytreebutisreallyjustanillusionthattakesalmostnospace.Ifwe
restrictoperationsonthecopytoaddingorremoving(unlinking)files—i.e.,
neverchangingoneinplace—thentheillusionofafullcopyiscomplete.To
theend-user,theonlydifferencesarethattheillusion-copytakesalmostno
disk space and almost no time to generate.
Wecancombine rsync and cp-al tocreatewhatappeartobemultiplefull
backups of a filesystem without taking multiple disks’ worth of space, as in:
rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete source_directory/ backup.0/
Iftheabovecommandsarerunonceeveryday,then backup.0, backup.1,
backup.2,and backup.3 willappeartoeachbeafullbackupof source_
directory/ asitappearedtoday,yesterday,twodaysago,andthreedaysago,
respectively—complete,exceptthatpermissionsandownershipsinold80 | Backups
#42 Automated Snapshot-Style Incremental Backups with rsync
HACK
snapshotswillgettheirmostrecentvalues(thankstoJ.W.Schultzforpoint-
ingthisout).Inreality,theextrastoragewillbeequaltothecurrentsizeof
source_directory/ plusthetotalsizeofthechangesoverthelastthreedays—
exactlythesamespacethatafullplusdailyincrementalbackupwith dump
or tar would have taken.
Thismethodis much betterfornetwork-basedbackups,sinceit’sonlynec-
essarytodoafullbackuponce,insteadofonceperweek.Thereafter,only
thechangesneedtobecopied.Unfortunately,youcan’t rsync toatape;
you’ll still need dump or tar for that.
Ifyouhaveasparemachine,evenaverylow-endone,youcanturnitintoa
dedicatedbackupserver.Makeitstandalone,andkeepitinaphysicallysep-
arateplace—anotherroomorevenanotherbuilding.Disableeverysingle
remoteserviceonthebackupserver,andconnectitonlytoadedicatednet-
work interface on the source machine.
Youcanthenperformbackupsfromthismachineusingrsyncoverssh(see
“KeepingPartsofFilesystemsinsyncwithrsync” [Hack#41]),andexportthe
backupsbacktotheoriginalmachineviaread-onlyNFS.Thenuserscanget
tothesnapshotsthemselves(withoutneedingsysadminintervention)and
can’t possibly change or delete them.
I’dconsiderthis“prettygood”protection,butifyou’re(wisely)paranoid,or
yourjobisontheline,buildtwobackupservers.Thenyoucanmakesure
that at least one of them is always offline.
Extensions: Hourly, Daily, and Weekly Snapshots
Withalittlebitoftweaking,youcanmakemultiple-levelrotatingsnap-
shots.Onmysystem,forexample,Ikeepthelastfour“hourly”snapshots
(whicharetakeneveryfourhours)aswellasthelastthree“daily”snap-
shots(whicharetakenatmidnighteveryday).Youmightalsowanttokeep
weeklyorevenmonthlysnapshotstoo,dependinguponyourneedsand
your available space.
Ikeeponescriptthatrunseveryfourhourstomakeandrotatehourlysnap-
shots,andanotherscriptthatrunsonceadayrotatethedailysnapshots.
Thereisnoneedtouse rsync forthehigher-levelsnapshots;just cp-al from
the appropriate hourly one.
Tomaketheautomaticsnapshotshappen,Ihaveaddedthefollowinglines
to root’s crontab:
0 */4 * * * /usr/local/bin/make_snapshot.sh
0 13 * * * /usr/local/bin/daily_snapshot_rotate.sh
Download from Wow! eBook <www.wowebook.com>Automated Snapshot-Style Incremental Backups with rsync#42
Backups | 81
HACK
Theycause make_snapshot.sh toberuneveryfourhoursonthehourand
daily_snapshot_rotate.sh toberuneverydayat13:00(thatis,1:00PM).
Those scripts are included below.
Listing: make_snapshot.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# RCS info: $Id$
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------
# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;
MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;
RSYNC=/usr/bin/rsync;
# ------------- file locations -----------------------------------------
MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_Rw=/root/snapshot;
EXCLUDES=/usr/local/etc/backup_exclude;
# ------------- the script itself --------------------------------------
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting...";
exit; } fi
# attempt to remount the Rw mount point as Rw; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readwrite";
exit;
}
fi;82 | Backups
#42 Automated Snapshot-Style Incremental Backups with rsync
HACK
# rotating snapshots of /home (fixme: this should be more general)
# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_Rw/home/hourly.3 ] ; then \
$RM -rf $SNAPSHOT_Rw/home/hourly.3 ; \
fi ;
# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_Rw/home/hourly.2 ] ; then \
$MV $SNAPSHOT_Rw/home/hourly.2 $SNAPSHOT_Rw/home/hourly.3 ; \
fi;
if [ -d $SNAPSHOT_Rw/home/hourly.1 ] ; then \
$MV $SNAPSHOT_Rw/home/hourly.1 $SNAPSHOT_Rw/home/hourly.2 ; \
fi;
# step 3: make a hard-link-only (except for dirs) copy of the latest
snapshot,
# if that exists
if [ -d $SNAPSHOT_Rw/home/hourly.0 ] ; then \
$CP -al $SNAPSHOT_Rw/home/hourly.0 $SNAPSHOT_Rw/home/hourly.1 ; \
fi;
# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC \
-va --delete --delete-excluded \
--exclude-from="$EXCLUDES" \
/home/ $SNAPSHOT_Rw/home/hourly.0 ;
# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_Rw/home/hourly.0 ;
# and thats it for home.
# now remount the Rw snapshot mountpoint as readonly
$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readonly";
exit;
} fi;
Ifyounoticeabove,Ihaveaddedanexcludeslisttothe rsync call.Thisis
justtopreventthesystemfrombackingupgarbagelikewebbrowsercaches,
whichchangefrequently(sothey’dtakeupspaceineverysnapshot)but
would be no loss if they were accidentally destroyed.Automated Snapshot-Style Incremental Backups with rsync#42
Backups | 83
HACK
Listing: Daily_snapshot_rotate.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: daily snapshots
# ----------------------------------------------------------------------
# RCS info: $Id: daily_snapshot_rotate.sh,v 1.2 2002/03/25 21:53:27 mrubel
Exp $
# ----------------------------------------------------------------------
# intended to be run daily as a cron job when hourly.3 contains the
# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
# ----------------------------------------------------------------------
# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;
MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
cp=/bin/cp;
# ------------- file locations -----------------------------------------
MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_Rw=/root/snapshot;
# ------------- the script itself --------------------------------------
# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting...";
exit; } fi
# attempt to remount the Rw mount point as Rw; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readwrite";
exit;
}
fi;
# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_Rw/home/daily.2 ] ; then \
$RM -rf $SNAPSHOT_Rw/home/daily.2 ; \
fi ;
# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_Rw/home/daily.1 ] ; then \
$MV $SNAPSHOT_Rw/home/daily.1 $SNAPSHOT_Rw/home/daily.2 ; \
fi;
if [ -d $SNAPSHOT_Rw/home/daily.0 ] ; then \84 | Backups
#43 Working with ISOs and CDR/CDRWs
HACK
$MV $SNAPSHOT_Rw/home/daily.0 $SNAPSHOT_Rw/home/daily.1; \
fi;
# step 3: make a hard-link-only (except for dirs) copy of
# hourly.3, assuming that exists, into daily.0
if [ -d $SNAPSHOT_Rw/home/hourly.3 ] ; then \
$cp -al $SNAPSHOT_Rw/home/hourly.3 $SNAPSHOT_Rw/home/daily.0 ; \
fi;
# note: do *not* update the mtime of daily.0; it will reflect
# when hourly.3 was made, which should be correct.
# now remount the Rw snapshot mountpoint as readonly
$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readonly";
exit;
} fi;
Sample Output of ls -l /snapshot/home
total 28
drwxr-xr-x 12 root root 4096 Mar 28 00:00 daily.0
drwxr-xr-x 12 root root 4096 Mar 27 00:00 daily.1
drwxr-xr-x 12 root root 4096 Mar 26 00:00 daily.2
drwxr-xr-x 12 root root 4096 Mar 28 16:00 hourly.0
drwxr-xr-x 12 root root 4096 Mar 28 12:00 hourly.1
drwxr-xr-x 12 root root 4096 Mar 28 08:00 hourly.2
drwxr-xr-x 12 root root 4096 Mar 28 04:00 hourly.3
Noticethatthecontentsofeachofthesubdirectoriesof /snapshot/home/ isa
completeimageof /home atthetimethesnapshotwasmade.Despitethe w
inthedirectoryaccesspermissions,noone—notevenroot—canwriteto
this directory; it’s mounted read-only.
See also:
•“Keeping Parts of Filesystems in sync with rsync” [Hack #41]
•See the version at http://www.mikerubel.org/computers/rsync_snapshots/)
HACK
#43
Working with ISOs and CDR/CDRWs Hack #43
The command line makes working with ISOs a snap
ThereareanumberofgraphicalCD-ROMutilitiesforLinux.Mostofthese
aresimplyfront-endsthatcallcommandlinetoolstodotheactualworkof
buildingISOsandburningdisks.Andgraphicaltoolsaren’tmuchhelpif
you’reworkingonserversthatdon’thaveaconsoleattached,unlessyouareWorking with ISOs and CDR/CDRWs#43
Backups | 85
HACK
runningXremotely,perhapsover ssh (asin“Xoverssh” [Hack#70]).Withthe
mkisofs and cdrecord utilitiesinstalled,workingwithISOimagesfromthe
command line is very straightforward.
Ifyou’veneverheardofanISO,it’sslangforanimageofanISO9660filesys-
tem.TheISO9660format(alongwithacoupleofcommonextensions)is
the common format of data CD-ROMs.
To make an ISO image to prepare for burning in a CD burner, use mkisofs:
mkisofs -r /home/rob/ > /tmp/rob-home.iso
The -r tells mkisofs tobuildRockRidgeextensionsintotheresultingimage
file.Thismeansthatlongfilenamesandfilepermissionswillbepreserved
whenthediskismountedonsystemsthatsupportRockRidge.Linuxhas
greatsupportforRR,andusing -r generallymakesyourlifemucheasierwhen
buildingdisksdesignedtobeusedwithLinux.AnyonecanmakeanISOof
filestheyhavereadaccessto;youdon’thavetoberoottorun mkisofs.Butfor
the rest of the commands in this hack, you’ll likely need root privileges.
Notethat mkisofs storesthecontentsofthedirectoriesyouspecifyonthe
commandline,notthedirectoriesthemselves.Intheaboveexample,the
ISOwillhave /home/rob/ mappedtoits /,withthecontentsof /home/rob/
filling the root of the CD.
If you need to make an ISO from an existing data CD, try this:
# dd if=/dev/cdrom of=image.iso
ThiswillcreateanISOimageoftheCD,inallglorious650+MB,sobesure
youhaveplentyofspacebeforetryingthis.Youmaybeabletomakethisgo
faster by manipulating the bs parameter of dd:
# dd if=/dev/cdrom of=image.iso bs=10k
TheoptimalsettingisusuallydependentonyourdriveandIDEcontroller,
so try a couple of values to see what works best on your system.
TomountanISOthatyou’vemade(eitherwith mkisofs orstraightfrom dd),
try this:
# mkdir /mnt/iso
# mount -o loop,ro -t iso9660 ./image.iso /mnt/iso
Ifthe mount commandcomplainsaboutCouldnotfindanyloopdevice,
you might need to load the loopback driver:
# modprobe loop
OncetheISOismounted,youshouldbeableto cd/mnt/iso andlook
aroundatthefilesystemcontainedintheimage.Ifyoufindproblemswith
it, simply umount /mnt/iso, remove the image file, and try again.86 | Backups
#44 Burning a CD Without Creating an ISO File
HACK
When you’re ready to burn an ISO to CDR, try something like this:
# cdrecord -v speed=12 dev=0,0,0 -data image.iso
Youshouldspecifythewritingspeedofyourburner(orslower)inthe
speed= option.IfyouneedtoeraseaCDRWbeforeburningtheISO,try
passing the blank= parameter:
# cdrecord -v speed=12 dev=0,0,0 blank=fast -data image.iso
GettingaburnerrunningunderLinuxisn’tashardasitusedtobe,thanks
tothe ide-scsi driver.ThisisakernelmodulethatmakesIDE(andother)CD
burners appear to be SCSI burners, which are much easier to program.
See also:
•ForhelpongettingyourburnerrunningunderLinux,checkouttheCD
WritingHOWTOavailableatTheLinuxDocumentationProjectat
http://www.tldp.org/HOWTO/CD-Writing-HOWTO.html
HACK
#44
Burning a CD Without Creating an ISO File Hack #44
Create a CD from another CD, the live filesystem, or even an http download
ThesafestmethodformakingacopyofaCDistofirstmakeanISOand
thenburntheISO(asin“WorkingwithISOsandCDR/CDRWs” [Hack#43]).
Butsometimesyoudon’thavethespace(ortime)fortheinterimstepof
making a copy.
Ifyouhaveafastenoughmachine,youcanusuallyburnstraightfromoneCD
toanother.ThisusuallyworksbestwhenthesourceCDandtheburnerareon
separate device chains (like primary and secondary IDE, or IDE and SCSI).
To make a real-time copy of a CD, give this a try:
# dd if=/dev/hdb | cdrecord -v speed=12 dev=0,0,0 fs=8m -data -
The-argumentto cdrecord meansthatthedatatrackshouldbereadfrom
STDINinsteadoffromafile.The dd lineisfeedingthe cdrecord pipewitha
copyoftheCDintheslavedriveontheprimaryIDEchain(hdb).The fs=8m
parametermakesthewriteFIFOabitbigger,tohelpoffsetanymomentary
pipelinehiccups.Aslongasyourbusisuptothetask(andyourmachine
isn’t otherwise too occupied) then this method will work fine.
Likewise,thereisnorealneedtomakeanISObeforeburningacopyofdata
from the filesystem. Give this a try:
# mkisofs -r /home/ftp/ | cdrecord -v speed=12 dev=0,0,0 fs=8m -data -Burning a CD Without Creating an ISO File#44
Backups | 87
HACK
Likethe dd above, mkisofs writestoSTDOUTbydefault.Thisisthenfedto
STDINofthe cdrecord process,burningtheISOasitiscreated,inrealtime.
ThissavestheneedtokeepacopyoftheISOfilelyingaroundyourfilesys-
tem.Bewarned,ifyourdatasourceisgreaterthanthesizeofyourCDR
(somewherebetween650and700MB)thenyou’llendupwithacoasterbut
not a usable disk.
Get an idea of how much space you’ll need with du first:
# du -hs /home/ftp/
412M /home/ftp
Perfect. A 412M ISO will fit nicely.
AnythingthatcanprintISOdatatoSTDOUTisacandidatefortheleft-
hand side of a pipeline. How about doing a real-time network burn?
root@catlin:~# mkisofs -r backup/ \
| ssh florian "cdrecord -v speed=12 dev=0,0,0 fs=8m -data -"
Or copying a local CD to a remote burner over the network?
root@catlin:~# dd if=/dev/cdrom \
| ssh florian "cdrecord -v speed=12 dev=0,0,0 fs=8m -data -"
Or even downloading an ISO and burning it, all in one pass?
# curl http://my.server.com/slackware-8.1-install.iso \
| cdrecord -v speed=0 dev=0,0,0 fs=8m -data -
Iwouldn’trecommendattemptingtodoanetworkburnoverwireless;
you’llwantanice,solid100MbpsEthernetcableforthisjob.Andthedown-
loadexamplemightlooksilly,butitdoesillustratethepoweroftheUnix
pipeline:anyprogramcanbepluggedintonearlyanyother,tomakethings
happen that the designer of either program likely hadn’t ever thought.88
Chapter4 CHAPTER FOUR
Networking
Hacks #45–53
Therewasonceatimewhenanetworkadminwasapersonwhospentallof
histimetryingtofigureouthowtomakemachinestalktoeachotherovera
network.Itseemsthatlately,muchofanetworkadmins’timeisspenttrying
tofigureouthowtorestrictaccesstotheirmachinesviathenetwork,thus
keepingoutundesirableswhilestillallowinglegitimatetraffictopassthrough.
Fortunately,thenetfilterfirewallinLinuxprovidesaveryflexibleinterface
tothekernel’snetworkingdecisions.Usingthe iptables command,youcan
createfirewallrulesthatletyoucreatearichandveryflexibleaccesspolicy.
Itcannotonlymatchpacketsbasedonport,interfaceandMACaddresses,
butalsoondatacontainedwithinthepacketandevenbytheratethatpack-
etsarereceived.Thisinformationcanbeusedtohelpweedoutallsortsof
attempted attacks, from port floods to virii.
Butlockingusersoutisn’tnearlyasmuchfunasconnectinguserstogether.
Afterallthewholepointofacomputernetworkistoallowpeopletocom-
municatewitheachother!We’lltakealookatsomemoreunusualmethods
forcontrollingtheflowofnetworktraffic,fromtheremoteportforwarding
tovariousformsofIPtunnelling.Bythetimewe’veexploredIPencapsula-
tionanduserspacetunnelslike vtun,we’llseehowitispossibletobuild
networksontopoftheInternetthatbehaveinallsortsofunexpectedand
surprisingly useful ways.
HACK
#45
CreatingaFirewallfromtheCommandLineof
any Server Hack #45
You don’t have to have a dedicated firewall to benefit from using iptables
Thenetfilterfirewall(availableinLinux2.4andlater)allowsforveryflexi-
blefirewallmanipulationfromthecommandline.Using iptables cantakea
whiletogetusedto,butitallowsforaveryexpressivesyntaxthatletsyou
create complex (and hopefully useful ;) firewall rules.Creating a Firewall from the Command Line of any Server#45
Networking | 89
HACK
Evenifyourmachineisn’ta“real”firewall(thatis,itonlyhasonenetwork
interfaceandisn’tprotectingothermachines)thefilterfunctionalitycanbe
veryuseful.Supposeyouwanttoallowtelnetaccesstothismachine(justin
casesomethinghappensto ssh oritslibraries)butdon’twanttopermititfrom
justanywhereontheNet.Youcoulduseatcpwrapper(bypopulating /etc/
hosts.allow and /etc/hosts.deny,andsettingup /etc/inetd.conf appropriately).
Or, you could use iptables with a line like this:
iptables -A INPUT -t filter -s ! 208.201.239.36 -p tcp --dport 23 -j DROP
Generally,mostpeoplewanttopermitunrestrictedaccessfromtrusted
hosts,blockallaccessfromknownproblemhosts,andallowsomethingin
betweenforeveryoneelse.Hereisonemethodforusingawhitelist,black-
list, and restricted port policy simultaneously.
#!/bin/sh
#
# A simple firewall initialization script
#
WHITELIST=/usr/local/etc/whitelist.txt
BLACKLIST=/usr/local/etc/blacklist.txt
ALLOWED="22 25 80 443"
#
# Drop all existing filter rules
#
iptables -F
#
# First, run through $WHITELIST, accepting all traffic from the hosts and
networks
# contained therein.
#
for x in `grep -v ^# $WHITELIST | awk '{print $1}'`; do
echo "Permitting $x..."
iptables -A INPUT -t filter -s $x -j ACCEPT
done
#
# Now run through $BLACKLIST, dropping all traffic from the hosts and
networks
# contained therein.
#
for x in `grep -v ^# $BLACKLIST | awk '{print $1}'`; do
echo "Blocking $x..."
iptables -A INPUT -t filter -s $x -j DROP
done
#
# Next, the permitted ports: What will we accept from hosts not appearing
# on the blacklist?
#
Download from Wow! eBook <www.wowebook.com>90 | Networking
#45 Creating a Firewall from the Command Line of any Server
HACK
for port in $ALLOWED; do
echo "Accepting port $port..."
iptables -A INPUT -t filter -p tcp --dport $port -j ACCEPT
done
#
# Finally, unless it's mentioned above, and it's an inbound startup request,
# just drop it.
#
iptables -A INPUT -t filter -p tcp --syn -j DROP
Besuretospecifyalloftheportsyou’dliketoincludeinthe$ALLOWED
variableatthetopofthescript.Ifyouforgettoinclude22,youwon’tbe
able to ssh into the box!
The /usr/local/etc/blacklist.txt fileispopulatedwithIPaddresses,host
names, and networks like this:
1.2.3.4 # Portscanned on 8/15/02
7.8.9.0/24 # Who knows what evil lurks therein
r00tb0y.script-kiddie.coop # $0 s0rR33 u 31337 h4x0r!
Likewise, /usr/local/etc/whitelist.txt containsthe“goodguys”thatshouldbe
permitted no matter what the other rules specify:
11.22.33.44 # My workstation
208.201.239.0/26 # the local network
Sincewe’reonlygrabbinglinesthatdon’tstartwith#,youcancomment
outanentirelineifyouneedto.Thenexttimeyourunthescript,anycom-
mentedentrieswillbeignored.Werunan iptables-F atthebeginningto
flushallexistingfilterentries,soyoucansimplyrunthescriptagainwhen
youmakechangesto blacklist.txt, whitelist.txt,ortheportsspecifiedin
$ALLOWED.
AlsonotethatthisscriptonlyallowsforTCPconnections.Ifyouneedto
alsosupportUDP,ICMP,orsomeotherprotocol,runanotherpassjustlike
the$ALLOWEDforloop,butincludeyouradditionalportsandprotocols
(passing -p udp or -p icmp to iptables, for example).
Becarefulaboutusingwhitelists.AnyIPsornetworksappearingonthislist
willbepermittedtoaccessallportsonyourmachine.Insomecircum-
stances,aclevermiscreantmaybeabletosendforgedpacketsapparently
originatingfromoneofthoseIPs,iftheycanfindoutaheadoftime(orlogi-
callydeduce)whatIPsappearonyourwhitelist.Thiskindofattackisdiffi-
culttoperform,butitispossible.Ifyouareparticularlyparanoid,youmight
onlyallowwhitelistaddressesfromnetworksthataren’troutableoverthe
Internet but are used on your internal network.Simple IP Masquerading#46
Networking | 91
HACK
Itisextremelyusefultohaveconsoleaccesswhileworkingwithnewfire-
wallrules(youcan’tlockyourselfoutoftheconsolewith iptables!)Ifyou
getconfusedaboutwhereyouarewhenworkingwith iptables,remember
thatyoucanalwayslistoutallruleswith iptables-L,andstartoverbyissu-
ing iptables-F.If iptables-L seemstohang,try iptables-L-n toshowthe
ruleswithoutdoinganyDNSresolution—yourrulesmightaccidentallybe
prohibiting DNS requests.
Youcandoalotwithsimplefiltering,butthere’smuchmoreto iptables
than just the filter target.
See also:
•Netfilter HOWTO at www.netfilter.org
•“iptables Tips & Tricks” [Hack #47]
HACK
#46
Simple IP Masquerading Hack #46
Set up NAT on your gateway in ten seconds
IfyouhaveaprivatenetworkthatneedstoshareanInternetconnection
withoneIPaddress,you’llwanttouseIPMasqueradingonyourgateway
machine. Luckily, with iptables this is a simple two-liner:
# echo "1" > /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o $EXT_IFACE -j MASQUERADE
where$EXT_IFACEistheoutsideinterfaceofyourgateway.Nowany
machinesthatresideonanetworkonanyoftheotherinterfacesinyour
gatewaywillbeableto“getout”totheInternet.AsfarastheNetiscon-
cerned, all traffic originates from your gateway’s external IP address.
Therewasatimewhenonehadtoworryaboutmiscreantsontheexternal
networksendingforgedpacketstoyourgateway,claimingtobeoriginating
fromtheinternalnetwork.Thesepacketswouldobliginglybemasqueraded
bythekernel,andleaveyournetworkasiftheywerelegitimatetraffic.This
madeitpossibleforanyonetolaunchattacksthatapparentlyoriginated
from your network, making very bad times for the hapless gateway owner.
Isaythatthiswasaproblem,becauserecentkernelsgiveyouafreefirewall
ruletodealwithexactlythisproblem,called rp_filter.With rp_filter
enabled,ifapacketthatarrivesonaninterfacehasasourceaddressthat
doesn’tmatchthecorrespondingroutingtableentry,itisdropped.This
effectivelypreventsIPspoofingandallowssimple(andsafe)masquerading
with the example above.92 | Networking
#47 iptables Tips & Tricks
HACK
Intheveryunlikelyeventthat rp_filter iscausingproblemsforyou,youcan
deactivate it very easily:
# echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter
See also:
•“iptables Tips & Tricks” [Hack #47]
HACK
#47
iptables Tips & Tricks Hack #47
Make your firewall do far more than filter packets with iptables
iptables isthenextgenerationoffirewallsoftwareforthenetfilterproject.It
providesallofthefunctionalityofitspredecessor,ipchains,inadditionto
supportforstatefulfirewalling. iptables alsosupportsaframeworkfor
extendingitscapabilitieswithloadablemodules.Hereareafewtricksyou
canusewiththebasedistributionof iptables,aswellassomeoftheextensi-
ble modules available for iptables.
Fortheseexamples,we’llassumethatthefollowingenvironmentvariables
are already set:
$EXT_IFACE
The external (public) interface of the firewall
$INT_IFACE
The inside (private) interface of the firewall
$DEST_IP
The ultimate requested destination of this packet
Youcanuse iptables tolimitnewinboundTCPpacketstopreventaDenial
of Service attack. This is accomplished with the following rules:
# Create syn-flood chain for detecting Denial of Service attacks
iptables -t nat -N syn-flood
# Limit 12 connections per second (burst to 24)
iptables -t nat -A syn-flood -m limit --limit 12/s --limit-burst 24 \
-j RETURN
iptables -t nat -A syn-flood -j DROP
# Check for DoS attack
iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -p tcp --syn \
-j syn-flood
TheseruleslimitnewinboundTCPconnections(packetswithSYNbitset)
to 12 per second after 24 connections per second have been seen.
Using iptables,atransparentSquidproxycanbesetup.Thiswilltranspar-
entlycacheandlogalloutboundHTTPrequeststotheInternet.Itrequiresiptables Tips & Tricks#47
Networking | 93
HACK
nomodificationtotheuser’sbrowserandisusefulforblockingunwanted
content.Thisisaccomplishedwiththefollowing iptables ruleatthetopof
the PREROUTING chain:
# Setup transparent Squid proxy for internal network
#
# For details on setting up Squid, see:
# http://www.linuxdoc.org/HOWTO/mini/TransparentProxy.html
#
iptables -t nat -A PREROUTING -i $INT_IFACE -p tcp --dport 80 \
-j REDIRECT --to-port 3128
ThisruleredirectsoutgoingrequestsonTCPport80toaSquidproxyrun-
ning on TCP port 3128 on the firewall.
ArbitraryTCPflagscanbematchedwithiptables.Thismeansyoucanblock
XMAS-tree (all flags set) and NULL packets with the following rules:
# DROP XMAS & NULL TCP packets
iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP
iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP
Advanced iptables Features
iptables hasintroducedseveraladvancedfirewallfeaturesthatareavailableby
patchingtheLinuxkernel.Thesepatchescanbeobtainedfrom http://www.
netfilter.org/ bydownloadingthepatch-o-maticversioncorrespondingtothe
iptables versionyouareusing.Patch-o-maticpatchesare iptables patchesthat
arenotyetavailableinthemainstreamLinuxkernel.Someofthepatchesare
experimental and should be used with caution.
Usingtheexperimentalnetfilter psd patch, iptables candetectandblock
inbound port scans with the following rule:
# DROP inbound port scans
iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -m psd -j DROP
Usingtheexperimentalnetfilter iplimit patch,iptablescanlimitthenumber
of connections received from a particular IP address with the following rule:
# DROP packets from hosts with more than 16 active connections
iptables -t nat -A PREROUTING -i $EXT_IFACE -p tcp --syn -d $DEST_IP -m
iplimit --iplimit-above 16 -j DROP
Oneofthemostpowerfulnetfilterpatchesallowsyoutomatchpackets
basedontheircontent.Theexperimentalstring-matchingpatchallowsyou
tofilteroutpacketsthatmatchacertainstring.Thisishelpfultofilterout
theCodeRedorNimdavirusesbeforetheyhityourwebserver.Thefollow-
ing rules achieve this:
# DROP HTTP packets related to CodeRed and Nimda viruses silently
iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
-m string --string "/default.ida?" -j DROP94 | Networking
#48 Forwarding TCP Ports to Arbitrary Machines
HACK
iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
-m string --string ".exe?/c+dir" -j DROP
iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
-m string --string ".exe?/c+tftp" -j DROP
Portforwardingisnownativeto iptables.Thenattableusesafeaturecalled
DestinationNATinthePREROUTINGchaintoaccomplishthis.Thefol-
lowingrulecanbeusedtoportforwardHTTPrequeststoasystem(10.0.0.3)
on the internal network:
# Use DNAT to port forward http
iptables -t nat -A PREROUTING ! -i $INT_IFACE -p tcp --destination-port \
80 -j DNAT --to 10.0.0.3:80
YoucanalsoportforwardUDPpackets.Ifyouportforwardtrafficforapar-
ticularport,youdonotneedtohaveacorrespondingruleintheINPUT
chaintoacceptinboundconnectionsonthatport.Thiswillonlyworkifthe
destinationisonanetworkonalocallyattachedinterface(thatis,notto
destinationsonforeignnetworks).Takealookattoolslike rinetd (“For-
wardingTCPPortstoArbitraryMachines” [Hack#48])or nportredird ifyou
need traffic to forward to remote networks.
IfyouportforwardyourHTTPrequeststoaninternalhost,youcanfilter
out the CodeRed virus in the FORWARD chain with this rule:
iptables -t filter -A FORWARD -p tcp --dport http \
-m string --string "/default.ida?" -j DROP
Using iptables canbechallengingatfirst,butitsflexibilitymakesitatre-
mendouslyusefultool.Ifyouevergetstuckwhiledevelopingyourruleset
(andyouwill),rememberthatyourtwobestfriendsare iptables-L-n and
tcpdump (maybe followed by a quick session with ethereal).
See also:
•“Forwarding TCP Ports to Arbitrary Machines” [Hack #48]
•The Linux Firewall HOWTO
HACK
#48
Forwarding TCP Ports to Arbitrary Machines Hack #48
Make non-local services appear to come from local ports
Aswesawin“iptablesTips&Tricks” [Hack#47],itissimpletoforwardTCP
andUDPportsfromafirewalltointernalhostsusing iptables.Butwhatif
youneedtoforwardtrafficfromarbitraryaddressestoamachinethatisn’t
even on your network? Try an application layer port forwarder, like rinetd.Forwarding TCP Ports to Arbitrary Machines#48
Networking | 95
HACK
Thissimplebitofcodeisacoupleofyearsoldbutissmall,efficient,and
perfectforjustthissortofproblem.Unpackthearchiveandsimplyrun
make,andyou’llbepresentedwithatiny rinetd binarythatwillletyou
forwardTCPportstoyourheart’scontent.Unfortunately,UDPportsaren’t
supported by rinetd.
The configuration file is dead simple:
[Source Address] [Source Port] [Destination Address] [Destination Port]
Eachporttobeforwardedisspecifiedonaseparateline.Thesourceand
destinationaddressescanbeeitherhostnamesorIPaddresses,andanIP
address of 0.0.0.0 binds rinetd to every available local IP:
0.0.0.0 80 some.othersite.gov 80
216.218.203.211 25 123.45.67.89 25
0.0.0.0 5353 my.shellserver.us 22
Savethefileto /etc/rinetd.conf,andcopy rinetd tosomewherehandy(like /usr/
local/sbin/, for example.) Then start it by simply running rinetd.
Thefirstexampleforwardsallwebtrafficdestinedforanylocaladdressto
some.othersite.gov.Notethatthiswillonlyworkifthereisn’tanotherpro-
cess (like Apache) already bound to local port 80.
ThenextforwardsinboundSMTPtrafficdestinedfor216.218.203.211to
themailserverat123.45.67.89(butdoesn’tinterferewithanySMTPagents
boundtootherlocalIPs).Thefinalexamplewillforwardanyinboundtraf-
ficonport5353tothe ssh serveron my.shellserver.us.Theseallworkwith-
outNAToranyspecialkernelconfiguration.Simplyrun rinetd,andit
daemonizes and starts listening on the ports you have specified.
Thisutilitycanreallyhelpeasethetransitionwhenrenumberingorphysi-
callyrelocatingservers,asservicescanappeartoremainupontheoriginal
IP(eventhoughtheyareactuallycomingfromanothernetworkentirely).
rinetd doesn’tevenneedtorunasroot,ifyou’reonlybindingtoportshigher
than1024.Therearealsoextensiveoptionsforprovidingaccesscontroland
keepinglogs.ThistinytooliswellworthhavinghandyforwhenTCPport
indirection is called.
See also:
• http://www.boutell.com/rinetd/
•“iptables Tips & Tricks” [Hack #47]96 | Networking
#49 Using Custom Chains in iptables
HACK
HACK
#49
Using Custom Chains in iptables Hack #49
Keep your firewall rules under control with custom chains
Bydefault,the iptables filtertableconsistsofthreechains:INPUT,FOR-
WARD,andOUTPUT.Youcanaddasmanycustomchainsasyouliketo
helpsimplifymanaginglargerulesets.Customchainsbehavejustasbuilt-in
chains,introducinglogicthatmustbepassedbeforetheultimatefateofa
packet is determined.
To create a new chain, use the -N switch:
root@mouse:~# iptables -N fun-filter
Youcanseewhichchainsaredefinedatanytimewiththestandard -L
switch:
root@mouse:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain fun-filter (0 references)
target prot opt source destination
Inordertomakeuseofyourcustomchain,you’llhavetojumptoitfrom
somewhere.Let’saddajumptothefun-filterchainwe’vejustcreated
straight from the INPUT chain:
root@mouse:~# iptables -t filter -A INPUT -j fun-filter
Nowyourcustomchaincangrowtoanysortofcomplexityyoulike.For
example,youmaywanttomatchpacketsbasedonthesourceMAC
address:
root@mouse:~# iptables -A fun-filter -m mac --mac-source 11:22:33:aa:bb:cc \
-j ACCEPT
root@mouse:~# iptables -A fun-filter -m mac --mac-source de:ad:be:ef:00:42 \
-j ACCEPT
root@mouse:~# iptables -A fun-filter -m mac --mac-source 00:22:44:fa:ca:de
-j REJECT --reject-with icmp-host-unreachable
root@mouse:~# iptables -A fun-filter -j RETURN
TheRETURNjumpattheendofthetablemakesprocessingresumebackin
thechainthatcalledthisone(inthiscase,backintheINPUTchain).Again,
show what all of your tables look like with the -L switch:Tunneling: IPIP Encapsulation#50
Networking | 97
HACK
root@mouse:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
fun-filter all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain fun-filter (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere MAC 11:22:33:AA:BB:CC
ACCEPT all -- anywhere anywhere MAC DE:AD:BE:EF:00:42
REJECT all -- anywhere anywhere MAC 00:22:44:FA:CA:DE reject-with icmp-host-
unreachable
RETURN all -- anywhere anywhere
Youcanjumpintoanynumberofcustomdefinedchainsandevenjump
betweenthem.Thishelpstoisolaterulesthatyou’redevelopingfromthe
standardsystempolicyrules,andenableanddisablethemeasily.Ifyou
wanttostopusingyourcustomchaintemporarily,youcansimplydeletethe
jump from the INPUT chain (rather than flushing the entire custom chain):
root@mouse:~# iptables -t filter -D INPUT -j fun-filter
If you decide to delete your custom chain, use -X:
root@mouse:~# iptables -X fun-filter
Notethattherecanbenoreferencestoyourcustomchainifyoutryto
deleteit;use -F toflushthechainfirstiftherearestillrulesreferringtoyour
chain.
Whenproperlymanaged,eventhemostcomplexiptablesrulesetscanbe
easily read, if you use intuitively named custom chains.
See also:
•“iptables Tips & Tricks” [Hack #47]
HACK
#50
Tunneling: IPIP Encapsulation Hack #50
IP tunneling with the Linux IPIP driver
IfyouhaveneverworkedwithIPtunnelingbefore,youmightwanttotakea
lookattheAdvancedRouterHOWTObeforecontinuing.Essentially,anIP
tunnelismuchlikeaVPN,exceptthatnoteveryIPtunnelinvolvesencryp-
tion.Amachinethatis“tunneled”intoanothernetworkhasavirtualinter-
faceconfiguredwithanIPaddressthatisn’tlocal,butexistsonaremote98 | Networking
#50 Tunneling: IPIP Encapsulation
HACK
network.Usually,all(ormost)networktrafficisrouteddownthistunnel,so
remoteclientsappeartoexistonthenetworkservices,ormoregenerally,to
connecttoanytwoprivatenetworkstogetherusingtheInternettocarrythe
tunnel traffic.
IfyouwanttoperformsimpleIP-within-IPtunnelingbetweentwo
machines,youmightwanttotryIPIP.Itisprobablythesimplesttunnelpro-
tocolavailableandwillalsoworkwith*BSD,Solaris,andevenWindows.
NotethatIPIPissimplyatunnelingprotocolanddoesnotinvolveanysort
ofencryption.Itisalsoonlycapableoftunnelingunicastpackets;ifyou
needtotunnelmulticasttraffic,takealookatGREtunnelingin“Tunnel-
ing: GRE Encapsulation” [Hack #51].
Beforewerushrightintoourfirsttunnel,you’llneedacopyoftheadvanced
routingtools(specificallythe ip utility).Youcangetthelatestauthoritative
copyat ftp://ftp.inr.ac.ru/ip-routing/.Bewarned,theadvancedroutingtools
aren’tespeciallyfriendly,buttheyallowyoutomanipulatenearlyanyfacet
of the Linux networking engine.
Assumethatyouhavetwoprivatenetworks(10.42.1.0/24and10.42.2.0/24)
andthatthesenetworksbothhavedirectInternetconnectivelyviaaLinux
routerateachnetwork.The“real”IPaddressofthefirstnetworkrouteris
240.101.83.2,andthe“real”IPofthesecondrouteris251.4.92.217.Thisisn’t
very difficult, so let’s jump right in.
First, load the kernel module on both routers:
# modprobe ipip
Next,onthefirstnetwork’srouter(onthe10.42.1.0/24network),dothe
following:
# ip tunnel add mytun mode ipip remote 251.4.92.217 \
local 240.101.83.2 ttl 255
# ifconfig mytun 10.42.1.1
# route add -net 10.42.2.0/24 dev mytun
And on the second network’s router (on the 10.42.2.0/24), reciprocate:
# ip tunnel add mytun mode ipip remote 240.101.83.2 \
local 251.4.92.217 ttl 255
# ifconfig tun10 10.42.2.1
# route add -net 10.42.1.0/24 dev mytun
Naturally,youcangivetheinterfaceamoremeaningfulnamethan
mytunifyoulike.Fromthefirstnetwork’srouter,youshouldbeableto
ping10.42.2.1,andfromthesecondnetworkrouter,youshouldbeable
toping10.42.1.1.Likewise,everymachineonthe10.42.1.0/24network
shouldbeabletoroutetoeverymachineonthe10.42.2.0/24network,
just as if the Interent weren’t even there.Tunneling: GRE Encapsulation#51
Networking | 99
HACK
Ifyou’rerunningaLinux2.2xkernel,you’reinluck:here’sashortcutthat
youcanusetoavoidhavingtousetheAdvancedRoutertoolspackageatall.
After loading the module, try these commands instead:
# ifconfig tun10 10.42.1.1 pointopoint 251.4.92.217
# route add -net 10.42.2.0/24 dev tun10
And on the second network’s router (on the 10.42.2.0/24):
# ifconfig tun10 10.42.2.1 pointopoint 240.101.83.2
# route add -net 10.42.1.0/24 dev tun10
That’s all there is to it.
Ifyoucanpingtheoppositerouterbutothermachinesonthenetworkdon’t
seemtobeabletopasstrafficbeyondtherouter,makesurethatbothrout-
ers are configured to forward packets between interfaces:
# echo "1" > /proc/sys/net/ipv4/ip_forward
Ifyouneedtoreachnetworksbeyond10.42.1.0and10.42.2.0,simplyadd
additional routeadd-net... lines.Thereisnoconfigurationneededonany
ofyournetworkhosts,aslongastheyhaveadefaultroutetotheirrespec-
tive router (which they definitely should, since it is their router, after all).
Tobringthetunneldown:Onbothrouters,bringdowntheinterfaceand
delete it, if you like:
# ifconfig mytun down
# ip tunnel del mytun
(or, in Linux 2.2):
# ifconfig tun10 down
Thekernelwillverypolitelycleanupyourroutingtableforyouwhenthe
interface goes away.
See also:
•Advanced Routing HOWTO, http://www.tldp.org/HOWTO/
Adv-Routing-HOWTO/
•Advanced Routing Tools (iproute2), ftp://ftp.inr.ac.ru/ip-routing/
HACK
#51
Tunneling: GRE Encapsulation Hack #51
IP tunnels with Generic Routing Encapsulation
GREstandsforGenericRoutingEncapsulation.LikeIPIPtunneling(see
“Tunneling:IPIPEncapsulation” [Hack#50]),GREisanunencryptedencapsula-
tionprotocol.ThemainadvantagesofusingGREinsteadofIPIPareitsup-
ports multicast packets, and it will interoperate with Cisco routers.
Download from Wow! eBook <www.wowebook.com>100 | Networking
#51 Tunneling: GRE Encapsulation
HACK
JustaswiththeIPIPtunnelinghack,we’llassumethatyouhavetwoprivate
networks(10.42.1.0/24and10.42.2.0/24)andthatthesenetworksboth
havedirectInternetconnectivityviaaLinuxrouterateachnetwork.The
“real”IPaddressofthefirstnetworkrouteris240.101.83.2,andthe“real”
IP of the second router is 251.4.92.217.
Again,aswithIPIPtunneling(“Tunneling:IPIPEncapsulation” [Hack#50]),
youwillneedacopyoftheadvancedroutingtoolspackage(thereisno
shortcutforGREtunnelsinLinux2.2thatI’vebeenabletofind).Onceyou
havethe iproute2 packageinstalled,we’llbeginbyloadingtheGREkernel
module on both routers:
# modprobe ip_gre
On the first network’s router, set up a new tunnel device:
# ip tunnel add gre0 mode gre remote 251.4.92.217 local 240.101.83.2 ttl 255
# ip addr add 10.42.1.254 dev gre0
# ip link set gre0 up
Notethatyoucancallthedeviceanythingyoulike; gre0 isjustanexample.
Also,that10.42.1.254addresscanbeanyavailableaddressonthefirstnet-
work,butshouldn’tbe10.42.1.1(theIPalreadyboundtoitsinternalinter-
face). Now, add your network routes via the new tunnel interface:
# ip route add 10.42.2.0/24 dev gre0
The first network is finished. Now for the second:
# ip tunnel add gre0 mode gre remote 240.101.83.2 local 251.4.92.217 ttl 255
# ip addr add 10.42.2.254 dev gre0
# ip link set gre0 up
# ip route add 10.42.1.0/24 dev gre0
Again,the10.42.2.254addresscanbeanyavailableaddressonthesecond
network.Feelfreetoaddasmany iprouteadd...devgre0 commandsas
you need.
That’sit!Youshouldnowbeabletopasspacketsbetweenthetwonet-
worksasiftheInternetdidn’texist.Atraceroutefromthefirstnetwork
shouldshowjustacoupleofhopstoanyhostinthesecondnetwork
(althoughyou’llprobablynoticeafairbitoflatencywhencrossingthe10.
42.2.254hop,unlessyou’rereallywellconnected).Ifyou’rehavingtrouble,
checkthenotesintheIPIPexampleanddon’tpanic.Yourbestfriendwhen
debuggingnewnetworkconfigurationsisprobablyapacketsnifferlike
tcpdump or ethereal.Runninga tcpdump'proto\icmp' onbothrouters
while pinging will give you a very detailed overview of what’s going on.
To bring the tunnel down, run this on both routers:
# ip link set gre0 down
# ip tunnel del gre0Using vtun over ssh to Circumvent NAT#52
Networking | 101
HACK
See also:
•Advanced Routing HOWTO, http://www.tldp.org/HOWTO/
Adv-Routing-HOWTO/
•Advanced Routing Tools (iproute2), ftp://ftp.inr.ac.ru/ip-routing/
HACK
#52
Using vtun over ssh to Circumvent NAT Hack #52
Connect two networks together using vtun and a single ssh connection
vtun isauserspacetunnelserver,allowingentirenetworkstobetunneledto
eachotherusingthe tun universaltunnelkerneldriver.Connectionscanbe
madedirectlyoverIPorevenoverPPPorserial.thistechniquecanbepartic-
ularlyusefulwhengeneralnetworkaccessisrestrictedbyaninterveningfire-
wall,asallIPtrafficwillbeencrytedandforwardedoverasingleTCPport
(that is, over a single ssh connection).
TheproceduredescribedbelowwillallowahostwithaprivateIPaddress
(10.42.4.6)tobringupanewtunnelinterfacewithareal,liveroutedIP
address(208.201.239.33)thatworksasexpected,asiftheprivatenetwork
weren’teventhere.We’lldothisbybringingupthetunnel,droppingthe
defaultroute,thenaddinganewdefaultrouteviatheotherendofthetunnel.
To begin with, here is the (pre-tunneled) configuration of the network.
root@client:~# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:02:2D:2A:27:EA
inet addr:10.42.3.2 Bcast:10.42.3.63 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:662 errors:0 dropped:0 overruns:0 frame:0
TX packets:733 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:105616 (103.1 Kb) TX bytes:74259 (72.5 Kb)
Interrupt:3 Base address:0x100
root@client:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.42.3.0 * 255.255.255.192 U 0 0 0 eth2
loopback * 255.0.0.0 U 0 0 0 lo
default 10.42.3.1 0.0.0.0 UG 0 0 0 eth2
Asyoucansee,ourlocalnetworkis10.42.3.0/26,ourIPis10.42.3.2,andour
defaultgatewayis10.42.3.1.Thisgatewayprovidesnetworkaddresstransla-
tion (NAT) to the internet. Here’s what our path looks like to yahoo.com:
root@client:~# traceroute -n yahoo.com
traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
1 10.42.3.1 2.848 ms 2.304 ms 2.915 ms102 | Networking
#52 Using vtun over ssh to Circumvent NAT
HACK
2 209.204.179.1 16.654 ms 16.052 ms 19.224 ms
3 208.201.224.194 20.112 ms 20.863 ms 18.238 ms
4 208.201.224.5 213.466 ms 338.259 ms 357.7 ms
5 206.24.221.217 20.743 ms 23.504 ms 24.192 ms
6 206.24.210.62 22.379 ms 30.948 ms 54.475 ms
7 206.24.226.104 94.263 ms 94.192 ms 91.825 ms
8 206.24.238.61 97.107 ms 91.005 ms 91.133 ms
9 206.24.238.26 95.443 ms 98.846 ms 100.055 ms
10 216.109.66.7 92.133 ms 97.419 ms 94.22 ms
11 216.33.98.19 99.491 ms 94.661 ms 100.002 ms
12 216.35.210.126 97.945 ms 93.608 ms 95.347 ms
13 64.58.77.41 98.607 ms 99.588 ms 97.816 ms
Inthisexample,we’llbeconnectingtoatunnelserverontheInternetat
208.201.239.5.IthastwospareliveIPaddresses(208.201.239.32and208.
201.239.33)tobeusedfortunneling.We’llrefertothatmachineasthe
server, and our local machine as the client.
Now,let’sgetthetunnelrunning.Tobeginwith,loadthe tun driveronboth
machines:
# modprobe tun
Itisworthnotingthatthe tun driverwillsometimesfailifthekernelversion
ontheserverandclientdon’tmatch.Forbestresults,usearecentkernel
(and the same version, e.g., 2.4.19) on both machines.
On the server machine, install this file to /usr/local/etc/vtund.conf:
options {
port 5000;
ifconfig /sbin/ifconfig;
route /sbin/route;
syslog auth;
}
default {
compress no;
speed 0;
}
home {
type tun;
proto tcp;
stat yes;
keepalive yes;
pass sHHH; # Password is REQUIRED.
up {
ifconfig "%% 208.201.239.32 pointopoint 208.201.239.33";
program /sbin/arp "-Ds 208.201.239.33 %% pub";
program /sbin/arp "-Ds 208.201.239.33 eth0 pub";Using vtun over ssh to Circumvent NAT#52
Networking | 103
HACK
route "add -net 10.42.0.0/16 gw 208.201.239.33";
};
down {
program /sbin/arp "-d 208.201.239.33 -i %%";
program /sbin/arp "-d 208.201.239.33 -i eth0";
route "del -net 10.42.0.0/16 gw 208.201.239.33";
};
}
and launch the vtund server with this command:
root@server:~# vtund -s
Now,you’llneeda vtund.conf fortheclientside.Trythisone,againin /usr/
local/etc/vtund.conf:
options {
port 5000;
ifconfig /sbin/ifconfig;
route /sbin/route;
}
default {
compress no;
speed 0;
}
home {
type tun;
proto tcp;
keepalive yes;
pass sHHH; # Password is REQUIRED.
up {
ifconfig "%% 208.201.239.33 pointopoint 208.201.239.32 arp";
route "add 208.201.239.5 gw 10.42.3.1";
route "del default";
route "add default gw 208.201.239.32";
};
down {
route "del default";
route "del 208.201.239.5 gw 10.42.3.1";
route "add default gw 10.42.3.1";
};
}
Finally, run this command on the client:
root@client:~# vtund -p home server104 | Networking
#52 Using vtun over ssh to Circumvent NAT
HACK
Presto!Younownotonlyhaveatunnelupbetweenclientandserver,but
haveaddedanewdefaultrouteviatheotherendofthetunnel.Takealook
at what happens when we traceroute to yahoo.com with the tunnel in place:
root@client:~# traceroute -n yahoo.com
traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
1 208.201.239.32 24.368 ms 28.019 ms 19.114 ms
2 208.201.239.1 21.677 ms 22.644 ms 23.489 ms
3 208.201.224.194 20.41 ms 22.997 ms 23.788 ms
4 208.201.224.5 26.496 ms 23.8 ms 25.752 ms
5 206.24.221.217 26.174 ms 28.077 ms 26.344 ms
6 206.24.210.62 26.484 ms 27.851 ms 25.015 ms
7 206.24.226.103 104.22 ms 114.278 ms 108.575 ms
8 206.24.238.57 99.978 ms 99.028 ms 100.976 ms
9 206.24.238.26 103.749 ms 101.416 ms 101.09 ms
10 216.109.66.132 102.426 ms 104.222 ms 98.675 ms
11 216.33.98.19 99.985 ms 99.618 ms 103.827 ms
12 216.35.210.126 104.075 ms 103.247 ms 106.398 ms
13 64.58.77.41 107.219 ms 106.285 ms 101.169 ms
Thismeansthatanyserverprocessesrunningonclientarenowfullyavail-
abletotheInternet,atIPaddress208.201.239.33.Thishashappenedall
withoutmakingasinglechange(e.g.,portforwarding)onthegateway10.
42.3.1.
Here’s what the new tunnel interface looks like on the client:
root@client:~# ifconfig tun0
tun0 Link encap:Point-to-Point Protocol
inet addr:208.201.239.33 P-t-P:208.201.239.32 Mask:255.255.255.255
UP POINTOPOINT RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39 errors:0 dropped:0 overruns:0 frame:0
TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:10
RX bytes:2220 (2.1 Kb) TX bytes:1560 (1.5 Kb)
andhere’stheupdatedroutingtable.Notethatwestillneedtokeepahost
routetothetunnelserver’sIPaddressviaourolddefaultgateway,other-
wise the tunnel traffic couldn’t get out:
root@client:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
208.201.239.5 10.42.3.1 255.255.255.255 UGH 0 0 0 eth2
208.201.239.32 * 255.255.255.255 UH 0 0 0 tun0
10.42.3.0 * 255.255.255.192 U 0 0 0 eth2
10.42.4.0 * 255.255.255.192 U 0 0 0 eth0
loopback * 255.0.0.0 U 0 0 0 lo
default 208.201.239.32 0.0.0.0 UG 0 0 0 tun0
Tobringdownthetunnel,simplykillthe vtund processontheclient.This
will restore all network settings back to their original state.Using vtun over ssh to Circumvent NAT#52
Networking | 105
HACK
Thismethodworksfine,ifyoutrust vtun tousestrongencryptionandtobe
freefromremoteexploits.Personally,Idon’tthinkyoucanbetooparanoid
whenitcomestomachinesconnectedtotheInternet.Touse vtun over ssh
(andthereforerelyonthestrongauthenticationandencryptionthat ssh pro-
vides)simplyforwardport5000onclienttothesameportonserver.Give
this a try:
root@client:~# ssh -f -N -c blowfish -C -L5000:localhost:5000 server
root@client:~# vtund -p home localhost
root@client:~# traceroute -n yahoo.com
traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
1 208.201.239.32 24.715 ms 31.713 ms 29.519 ms
2 208.201.239.1 28.389 ms 36.247 ms 28.879 ms
3 208.201.224.194 48.777 ms 28.602 ms 44.024 ms
4 208.201.224.5 38.788 ms 35.608 ms 35.72 ms
5 206.24.221.217 37.729 ms 38.821 ms 43.489 ms
6 206.24.210.62 39.577 ms 43.784 ms 34.711 ms
7 206.24.226.103 110.761 ms 111.246 ms 117.15 ms
8 206.24.238.57 112.569 ms 113.2 ms 111.773 ms
9 206.24.238.26 111.466 ms 123.051 ms 118.58 ms
10 216.109.66.132 113.79 ms 119.143 ms 109.934 ms
11 216.33.98.19 111.948 ms 117.959 ms 122.269 ms
12 216.35.210.126 113.472 ms 111.129 ms 118.079 ms
13 64.58.77.41 110.923 ms 110.733 ms 115.22 ms
Inordertodiscourageconnectionsto vtund onport5000oftheserver,add
a netfilter rule to drop connections from the outside world:
root@server:~# iptables -A INPUT -t filter -i eth0 -p tcp --dport 5000 -j
DROP
Thiswillallowlocalconnectionstogetthrough(sincetheyuseloopback),and
therefore require an ssh tunnel to server before accepting a connection.
Asyoucansee,thiscanbeanextremelyhandytooltohavearound.Inaddi-
tiontogivingliveIPaddressestomachinesbehindaNAT,youcaneffec-
tivelyconnectanytwonetworkstogetherifyoucanobtainasingle ssh
connection between them (originating from either direction).
Ifyourheadisswimmingfromthe vtund.conf configurationpreviously,orif
you’reterminallylazyanddon’twanttofigureoutwhattochangewhenset-
tingupyourownclient’s vtund.conf,takealookattheAutomatic vtund.conf
generator, in“Automatic vtund.conf Generator” [Hack #53].
Notes:
•Thesessionname(home intheaboveexample)mustmatchonthecli-
entANDserversides,oryou’llgetanambiguous“serverdisconnected”
message.
•Thesamegoesforthe password fieldinthe vtund.conf.Itmustbepresent
AND match on both sides, or the connection won’t work.106 | Networking
#53 Automatic vtund.conf Generator
HACK
•Ifyou’rehavingtroubleconnecting,makesureyou’reusingthesame
kernelversiononbothsides,andthattheserverisupandrunning(try
telnetserver5000 fromtheclientsidetoverifythattheserveris
happy).
•Trythedirectmethodfirst,thengetsshworkingonceyouarehappy
with your vtund.conf settings.
•Ifyou’restillhavingtrouble,check /etc/syslog.conf toseewhereyour
auth facilitymessagesaregoing,andwatchthatlogonboththeclient
and server when trying to connect.
See also:
• vtun’s homepage: http://vtun.sourceforge.net/
• /usr/src/linux/Documentation/networking/tuntap.txt
•man vtund; man vtund.conf
•“Automatic vtund.conf Generator” [Hack #53]
•“Creating a Firewall from the Command Line of any Server” [Hack #45]
•“Forwarding Ports over ssh” [Hack #71]
HACK
#53
Automatic vtund.conf Generator Hack #53
Generate a vtund.conf on the fly to match changing network conditions
Ifyou’vejustcomefrom“Tunneling:GREEncapsulation” [Hack#51],thisscript
will generate a working vtund.conf for the client side automatically.
Ifyouhaven’tread“Tunneling:GREEncapsulation” [Hack#51] (orifyou’ve
neverused vtun),thengobackandreaditbeforeattemptingtogrokthisbit
ofPerl.Essentially,itattemptstotaketheguessworkoutofchangingthe
routingtablearoundontheclientsidebyauto-detectingthedefaultgate-
way and building the vtund.conf accordingly.
Toconfigurethescript,takealookattheConfigurationsection.Thefirst
lineof$Configcontainstheaddresses,port,andsecretthatweusedin
“Tunneling:GREEncapsulation” [Hack#51].Thesecondistheresimplyasan
example of how to add more.
Torunthescript,eithercallitas vtundconfhome,orset$TunnelNameto
theonetowhichyouwanttodefault.Orbetteryet,makesymlinkstothe
script like this:
# ln -s vtundconf home
# ln -s vtundconf tunnel2
Then generate the appropriate vtund.conf by calling the symlink directly:
# vtundconf home > /usr/local/etc/vtund.confAutomatic vtund.conf Generator#53
Networking | 107
HACK
Youmightbewonderingwhyanyonewouldgotoallofthetroubletomake
ascripttogeneratea vtund.conf inthefirstplace.Onceyougetthesettings
right, you’ll never have to change them, right?
Well,usuallythatisthecase.ButconsiderthecaseofaLinuxlaptopthat
usesmanydifferentnetworksinthecourseoftheday(sayaDSLlineat
home,Ethernetatwork,andmaybeawirelessconnectionatthelocalcoffee
shop).Byrunning vtund.conf onceateachlocation,youwillhaveaworking
configurationinstantly,evenifyourIPandgatewayisassignedbyDHCP.
Thismakesitveryeasytogetupandrunningquicklywithalive,routableIP
address, regardless of the local network topology.
Incidentally, vtund and vtund.conf currentlyrunsgreatonLinux,FreeBSD,
OS X, Solaris, and a few others.
Listing: vtundconf
#!/usr/bin/perl -w
# vtund wrapper in need of a better name.
#
# (c)2002 Schuyler Erle & Rob Flickenger
#
################ CONFIGURATION
# If TunnelName is blank, the wrapper will look at @ARGV or $0.
#
# Config is TunnelName, LocalIP, RemoteIP, TunnelHost, TunnelPort, Secret
#
my $TunnelName = "";
my $Config = q{
home 208.201.239.33 208.201.239.32 208.201.239.5 5000 sHHH
tunnel2 10.0.1.100 10.0.1.1 192.168.1.4 6001 foobar
};
################ MAIN PROGRAM BEGINS HERE
use POSIX 'tmpnam';
use IO::File;
use File::Basename;
use strict;
# Where to find things...
#
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin";
my $IP_Match = '((?:\d{1,3}\.){3}\d{1,3})'; # match xxx.xxx.xxx.xxx
my $Ifconfig = "ifconfig -a";
my $Netstat = "netstat -rn";
my $Vtund = "/bin/echo";
my $Debug = 1;108 | Networking
#53 Automatic vtund.conf Generator
HACK
# Load the template from the data section.
#
my $template = join( "", <DATA> );
# Open a temp file -- adapted from Perl Cookbook, 1st Ed., sec. 7.5.
#
my ( $file, $name ) = ("", "");
$name = tmpnam() until $file = IO::File->new( $name, O_RDWR|O_CREAT|O_EXCL
);
END { unlink( $name ) or warn "Can't remove temporary file $name!\n"; }
# If no TunnelName is specified, use the first thing on the command line,
# or if there isn't one, the basename of the script.
# This allows users to symlink different tunnel names to the same script.
#
$TunnelName ||= shift(@ARGV) || basename($0);
die "Can't determine tunnel config to use!\n" unless $TunnelName;
# Parse config.
#
my ($LocalIP, $RemoteIP, $TunnelHost, $TunnelPort, $Secret);
for (split(/\r*\n+/, $Config)) {
my ($conf, @vars) = grep( $_ ne "", split( /\s+/ ));
next if not $conf or $conf =~ /^\s*#/o; # skip blank lines, comments
if ($conf eq $TunnelName) {
($LocalIP, $RemoteIP, $TunnelHost, $TunnelPort, $Secret) = @vars;
last;
}
}
die "Can't determine configuration for TunnelName '$TunnelName'!\n"
unless $RemoteIP and $TunnelHost and $TunnelPort;
# Find the default gateway.
#
my ( $GatewayIP, $ExternalDevice );
for (qx{ $Netstat }) {
# In both Linux and BSD, the gateway is the next thing on the line,
# and the interface is the last.
#
if ( /^(?:0.0.0.0|default)\s+(\S+)\s+.*?(\S+)\s*$/o ) {
$GatewayIP = $1;
$ExternalDevice = $2;
last;
}
}
die "Can't determine default gateway!\n" unless $GatewayIP and
$ExternalDevice;
# Figure out the LocalIP and LocalNetwork.
#
Download from Wow! eBook <www.wowebook.com>Automatic vtund.conf Generator#53
Networking | 109
HACK
my ( $LocalNetwork );
my ( $iface, $addr, $up, $network, $mask ) = "";
sub compute_netmask {
($addr, $mask) = @_;
# We have to mask $addr with $mask because linux /sbin/route
# complains if the network address doesn't match the netmask.
#
my @ip = split( /\./, $addr );
my @mask = split( /\./, $mask );
$ip[$_] = ($ip[$_] + 0) & ($mask[$_] + 0) for (0..$#ip);
$addr = join(".", @ip);
return $addr;
}
for (qx{ $Ifconfig }) {
last unless defined $_;
# If we got a new device, stash the previous one (if any).
if ( /^([^\s:]+)/o ) {
if ( $iface eq $ExternalDevice and $network and $up ) {
$LocalNetwork = $network;
last;
}
$iface = $1;
$up = 0;
}
# Get the network mask for the current interface.
if ( /addr:$IP_Match.*?mask:$IP_Match/io ) {
# Linux style ifconfig.
compute_netmask($1, $2);
$network = "$addr netmask $mask";
} elsif ( /inet $IP_Match.*?mask 0x([a-f0-9]{8})/io ) {
# BSD style ifconfig.
($addr, $mask) = ($1, $2);
$mask = join(".", map( hex $_, $mask =~ /(..)/gs ));
compute_netmask($addr, $mask);
$network = "$addr/$mask";
}
# Ignore interfaces that are loopback devices or aren't up.
$iface = "" if /\bLOOPBACK\b/o;
$up++ if /\bUP\b/o;
}
die "Can't determine local IP address!\n" unless $LocalIP and $LocalNetwork;
# Set OS dependent variables.
#
my ( $GW, $NET, $PTP );
if ( $^O eq "linux" ) {
$GW = "gw"; $PTP = "pointopoint"; $NET = "-net";
} else {110 | Networking
#53 Automatic vtund.conf Generator
HACK
$GW = $PTP = $NET = "";
}
# Parse the config template.
#
$template =~ s/(\$\w+)/$1/gee;
# Write the temp file and execute vtund.
#
if ($Debug) {
print $template;
} else {
print $file $template;
close $file;
system("$Vtund $name");
}
__DATA__
options {
port $TunnelPort;
ifconfig /sbin/ifconfig;
route /sbin/route;
}
default {
compress no;
speed 0;
}
$TunnelName { # 'mytunnel' should really be `basename $0` or some such
# for automagic config selection
type tun;
proto tcp;
keepalive yes;
pass $Secret;
up {
ifconfig "%% $LocalIP $PTP $RemoteIP arp";
route "add $TunnelHost $GW $GatewayIP";
route "delete default";
route "add default $GW $RemoteIP";
route "add $NET $LocalNetwork $GW $GatewayIP";
};
down {
ifconfig "%% down";
route "delete default";
route "delete $TunnelHost $GW $GatewayIP";
route "delete $NET $LocalNetwork";
route "add default $GW $GatewayIP";
};
}111
Chapter5 CHAPTER FIVE
Monitoring
Hacks #54–65
Itisdifficulttoknowhowtotunearunningsystemifyouhavenoideahowa
system“normally”runs.Byaskingcarefulquestionsofthesystem(andinter-
pretingtheanswerscorrectly),youcanavoidpokingaroundatvariablesin
the dark and make effective changes exactly where they need to be made.
Thisiswherelogfilescanbeyourbestfriend.Treatthemwell,andpay
themtheattentionthey’redue;youwilllearnvolumesabouthowyoursys-
temisbeingused.Butifyousimplyletthemfillupyourdisks,theycanbea
sourceofmuchconfusion.Withthepropertoolsandtechniques,yoursys-
temlogswillbeconciseanddetailedenoughtotellyouexactlywhatyou
need to know.
Butsometime,theinformationyou’reafterdoesn’tgetloggedanywhere,but
isexpressedintherunningLinuxsystem,eitherasapinpointcheckofsystem
resourcesordataonthenetworkitself.Incidentally,wewon’texaminefull-
blownsystemmonitoringandtrendingpackages(suchasNagiosorMRTG)
inthischapterbutinsteadwilllookatwaystointerrogateyoursystemtoget
specificinformationaboutwhat’sgoingon rightnow.We’llalsoseeacouple
ofwaysoftodetectpotentialproblemsbeforetheyhappenandevenhowto
automatically deal with catastrophic failures when they do occur.
HACK
#54
Steering syslog Hack #54
Make syslog work harder, and spend less time looking through huge log files
Thedefaultsysloginstallationonmanydistributionsdoesn’tdoaverygood
joboffilteringclassesofinformationintoseparatefiles.Ifyouseeajumble
ofmessagesfrom sendmail, sudo, bind,andothersystemservicesin /var/log/
messages, then you should probably review your /etc/syslog.conf.
Thereareanumberoffacilitiesandprioritiesonwhichsyslogcanfilter.For
easy reference, here they are:112 | Monitoring
#54 Steering syslog
HACK
Notethatapplicationsdecideforthemselvesatwhatfacilityandpriorityto
log(andthebestapplicationsletyouchoose),sotheymaynotalwaysbe
loggedasyouexpect.Here’sasample /etc/syslog.conf thatattemptstoshuf-
fle around what gets logged where:
auth.warning /var/log/auth
mail.err /var/log/maillog
kern.* /var/log/kernel
cron.crit /var/log/cron
*.err;mail.none /var/log/syslog
*.info;auth.none;mail.none /var/log/messages
#*.=debug /var/log/debug
local0.info /var/log/cluster
local1.err /var/log/spamerica
Alloftheabovelineswilllogthespecifiedpriority(orhigher)totherespec-
tivefile.Thespecialpriority none tellssyslognottobotherloggingthespeci-
fiedfacilityatall.The local0 through local7 facilitiesaresuppliedforuse
withyourownprograms,howeveryouseefit.Forexample,the /var/log/
spamerica filefillswith local1.err (orhigher)messagesthataregeneratedby
ourspamprocessingjob.It’snicetohavethosemessagesseparatefromthe
standard mail delivery log (in /var/log/maillog.)
Thatcommented *.=debug lineisusefulwhendebuggingdaemonizedser-
vices.Ittellssyslogtospecificallylogonlydebugprioritymessagesofany
facilityandgenerallyshouldn’tberunning(unlessyoudon’tmindfilling
yourdiskswithdebuglogs).Anotherapproachistologdebuginformationto
afifo.Thiswillmakedebuglogstakeupnospacebutwilldisappearunlessa
process is watching it. To log to a fifo, first create it in the filesystem:
# mkfifo -m 0664 /var/log/debug
Facilities Priorities
authdebug
auth-privinfo
cronnotice
daemonwarning
kernerr
lprcrit
mailalert
newsemerg
syslog
user
uucp
local0 - local7Steering syslog#54
Monitoring | 113
HACK
Then amend the debug line in syslog.conf to include a | like this:
*.=debug |/var/log/debug
Nowdebuginformationisconstantlyloggedtothefifo,andcanbeviewed
withacommandlike less-f/var/log/debug.Thisisalsohandytosetupif
youwantaprocesstoconstantlywatchallsystemmessagesandperhaps
notifyyouviaemailwhenacriticalsystemmessageisseen.Trymakingafifo
called /var/log/monitor, and add a rule like this to your syslog.conf:
*.* |/var/log/monitor
Noweverymessage(ateverypriority)ispassedtothe /var/log/monitor fifo,
andanyprocesswatchingitcanreactaccordingly,allwithouttakingupany
disk space.
Mark Who?
Do you notice a bunch of lines like this in /var/log/messages?
Dec 29 18:33:35 catlin -- MARK --
Dec 29 18:53:35 catlin -- MARK --
Dec 29 19:13:35 catlin -- MARK --
Dec 29 19:33:35 catlin -- MARK --
Dec 29 19:53:35 catlin -- MARK --
Dec 29 20:13:35 catlin -- MARK --
Dec 29 20:33:35 catlin -- MARK --
Dec 29 20:53:35 catlin -- MARK --
Dec 29 21:13:35 catlin -- MARK --
Thesearegeneratedbythemarkfunctionalityofsyslog,asawayof“touch-
ingbase”withthesystem,sothatyoucantheoreticallytellifsysloghas
unexpectedlydied.Mosttimes,thisonlyservestofillyourlogfiles,and
unlessyouarehavingproblemswithsyslog,youprobablydon’tneedit.To
turnthisoff,passthe -m0 switchtosyslogd(afterfirstkillinganyrunning
syslogd), like this:
# killall syslogd; /usr/sbin/syslogd -m 0
Remote Logging
Modernversionsofsyslogddisabletheabilitytoreceivelogsfromremote
machinesandforgoodreason.Withoutsafeguardsinplace,itisentirely
possibleforarandommiscreanttofillupyourdiskwithbogussyslogmes-
sages,asthereisnohostauthenticationavailableinsyslog.Still,itisvery
handytohaveacentralizedsyslog,particularlywhendealingwithacluster
of machines.
To turn on remote reception on the master, start syslogd with -r:
# /usr/sbin/syslogd -m 0 -r114 | Monitoring
#55 Watching Jobs with watch
HACK
On each of the machines you’d like to log from, add this to the syslog.conf:
*.* @master.syslog.host.com
(Naturally,withyourownhostnameorIPaddressafterthe @.)Itisavery
goodideatoprotectthesyslogportonyoursyslogserver.ItlistensonUDP
port 514 and is easily filtered with this iptables command:
# iptables -A INPUT -t filter -p udp --dport 514 -s ! $LOCAL_NET -j DROP
where$LOCAL_NETisthenetwork(orhost)thatyouwouldliketoreceive
sylog messages from (e.g., 192.168.1.0/24 or florian.nocat.net).
Onceyoursystemlogsarebetterorganized,itbecomesmucheasiertofind
theinformationyou’relookingfor.Eventhen,systemlogscangetabitover-
whelming.Forevenmorehelpdealingwithlogfiles,checkout“Colorized
Log Analysis in Your Terminal” [Hack #75].
HACK
#55
Watching Jobs with watch Hack #55
Use watch to repeatedly run any command, and show you the results
Ifyouhaveeverhadalong-runningbackgroundprocess,chancesareyou
areusedtorepeatedlycheckingtoseeifthecommandhasfinishedusing ps
and grep:
mc@escher:~$ ps ax |grep tar
10303 ? S 0:00 bash -c cd /; tar cf - home
10304 ? S 0:42 tar cf - home
mc@escher:~$ ps ax |grep tar
10303 ? S 0:00 bash -c cd /; tar cf - home
10304 ? S 0:43 tar cf - home
Ormaybeyou’re ftpingafileandforgetthe hash command(andforsomerea-
son,aren’tusing ncftp,scp, or wget or curl),soyouhavenoideahowfaralong
your transfer is. Naturally, you log in again and take a look at the file with ls:
mc@escher:~$ ls -l xfree86.tgz
-rw-r--r-- 1 rob users 12741812 Jun 13 2001 xfree86.tgz
mc@escher:~$ ls -l xfree86.tgz
-rw-r--r-- 1 rob users 12744523 Jun 13 2001 xfree86.tgz
Anytimeyoufindyourselfrunningacommandoverandoveragain,trythe
watch command.Itwillrepeatedlycycleoveranycommandyougiveit,and
whateverintervalyou’dlike(defaultingto2seconds).Itclearsthescreenon
each pass, making a nice, easy to read display.
mc@escher:~$ watch 'ps ax |grep tar'
Every 2s: ps ax |grep tar|grep -v pts/0 Fri Sep 6 00:22:01 2002
10303 ? S 0:00 bash -c cd /; tar cf - home
10304 ? S 0:42 tar cf - homeWhat’s Holding That Port Open?#56
Monitoring | 115
HACK
You’llonlyneedtoenclosethecommandinsinglequotesifyou’reusing
pipes(orotherspecialcharactersthatyoudon’twantinterpolatedbefore
watch runs). To specify a different time interval, try the -n switch:
mc@escher:~$ watch -n1 ls -l xfree86.tgz
Every 1s: ls -l xfree86.tgz Fri Sep 6 00:31:41 2002
-rw-r--r-- 1 rob users 12756042 Jun 13 2001 xfree86.tgz
Itwillevenhighlightthedifferencesoneachpass,makingchangesleapout
inreversetype.Trythe -d switchforthat(it’sespeciallyniceforwatching
theoutputofa netstat-a|grepESTAB,towatchnetworkconnections
comeandgo.Thinkof watch asyourobsessive-compulsivelittlebuddy.It
obsesses at your slightest whim, so you don’t have to. To exit, hit ^C.
See also:
•man watch
HACK
#56
What’s Holding That Port Open? Hack #56
Associate a process with the port it is bound to easily with netstat
GeneratingalistofnetworkportsthatareintheListenstateonaLinux
server is simple with netstat:
root@catlin:~# netstat -ln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 10.42.3.2:53 0.0.0.0:* LISTEN
tcp 0 0 10.42.4.6:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
udp 0 0 10.42.3.2:53 0.0.0.0:*
udp 0 0 10.42.4.6:53 0.0.0.0:*
udp 0 0 127.0.0.1:53 0.0.0.0:*
udp 0 0 0.0.0.0:67 0.0.0.0:*
raw 0 0 0.0.0.0:1 0.0.0.0:* 7
So,weseetheusualservices(awebserveronport80,DNSonport53, ssh on
port 22, dhcp on port 67), but what’s that process listening on 5280?
Findingoutwhichprogramsareactuallyboundtothoseportsissimplewith
recentversionsofnetstat.Aslongasyou’reroot,justaddthe -p switch(for
programs):
root@catlin:~# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name116 | Monitoring
#57 Checking On Open Files and Sockets with lsof
HACK
tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN 698/perl
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 217/httpd
tcp 0 0 10.42.3.2:53 0.0.0.0:* LISTEN 220/named
tcp 0 0 10.42.4.6:53 0.0.0.0:* LISTEN 220/named
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 220/named
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 200/sshd
udp 0 0 0.0.0.0:32768 0.0.0.0:* 220/named
udp 0 0 10.42.3.2:53 0.0.0.0:* 220/named
udp 0 0 10.42.4.6:53 0.0.0.0:* 220/named
udp 0 0 127.0.0.1:53 0.0.0.0:* 220/named
udp 0 0 0.0.0.0:67 0.0.0.0:* 222/dhcpd
raw 0 0 0.0.0.0:1 0.0.0.0:* 7 222/dhcpd
Ah,that’sbetter.PID698isaPerlprocessthatisboundtoport5280.We
now hunt it down with ps:
root@catlin:~# ps auwex |grep -w 698
nocat 698 0.0 2.0 5164 3840 ? S Aug25 0:00 /usr/bin/perl -w ./bin/gateway
PWD=/usr/local/nocat HOSTNAME=catlin.r
The psaweux showsusall(a)non-interactive(x)processeswithuserinfor-
mation(u)inwideformat(w)withsomeenvironmentbitsappended(e).
We then grep on word boundaries (-w) for the PID.
That’sbetter:nowweknowthatthenocatuserisinthe /usr/local/nocat/
running bin/gateway,aPerlprocessthatislisteningonport5280.Without
the -p switchon netstat,associatinganopenportwithaparticularprocessis
much trickier.
Incidentally,ifyou’renotroot,thenthesystemwon’tdisclosewhichpro-
grams are running on which ports. If you see an error like this:
(No info could be read for "-p": geteuid()=1000 but you should be root.)
thencomebackwhenyou’reroot!(Orsee“MakesudoWorkHarder” [Hack
#12] for how to sudo yourself into an amazing simulation of root.)
HACK
#57
Checking On Open Files and Sockets with
lsof Hack #57
Easilyseewhichfiles,directories,andsocketsyourrunningprocessesare
holding open
Haveyouevertriedto umount afilesystem,onlytofindthatsomeprocess
was still using it?
root@mouse:~# umount /mnt
umount: /mnt: device is busy
To quickly hunt down what processes are still using /mnt, try the lsof tool:
root@mouse:~# lsof /mnt
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 30951 rob cwd DIR 7,0 1024 2 /mntChecking On Open Files and Sockets with lsof#57
Monitoring | 117
HACK
Ah,apparentlyrobis cd’dto /mnt (sincehis bash processhasitsetasits
cwd). lsof willlistallopenfiles,directories,libraries,sockets,anddevices
associatedwithaparticularprocess.Intheaboveexample,wespecifieda
mountpointandhad lsof showustheassociatedprocesses.Todothe
reverse (show files associated with a PID), use the -p switch:
root@mouse:~# lsof -p 30563
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
inetd 30563 root cwd DIR 3,3 408 2 /
inetd 30563 root rtd DIR 3,3 408 2 /
inetd 30563 root txt REG 3,3 21432 39140 /usr/sbin/inetd
inetd 30563 root mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
inetd 30563 root mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
inetd 30563 root mem REG 3,3 19148 11708 /lib/libnss_db-2.2.so
inetd 30563 root mem REG 3,3 238649 11728 /lib/libnss_files-2.2.3.so
inetd 30563 root mem REG 3,3 483324 11710 /lib/libdb-3.1.so
inetd 30563 root 0u CHR 1,3 647 /dev/null
inetd 30563 root 1u CHR 1,3 647 /dev/null
inetd 30563 root 2u CHR 1,3 647 /dev/null
inetd 30563 root 4u IPv4 847222 TCP *:telnet (LISTEN)
inetd 30563 root 5u IPv4 560439 TCP *:cvspserver (LISTEN)
If you’d rather specify the process by name, use -c:
root@mouse:~# lsof -c syslogd
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
syslogd 25627 root cwd DIR 3,3 408 2 /
syslogd 25627 root rtd DIR 3,3 408 2 /
syslogd 25627 root txt REG 3,3 27060 5538 /usr/sbin/syslogd
syslogd 25627 root mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
syslogd 25627 root mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
syslogd 25627 root mem REG 3,3 238649 11728 /lib/libnss_files-2.2.3.so
syslogd 25627 root mem REG 3,3 75894 11719 /lib/libnss_dns-2.2.3.so
syslogd 25627 root mem REG 3,3 225681 11724 /lib/libresolv-2.2.3.so
syslogd 25627 root mem REG 3,3 19148 11708 /lib/libnss_db-2.2.so
syslogd 25627 root mem REG 3,3 483324 11710 /lib/libdb-3.1.so
syslogd 25627 root 0u unix 0xdcc2d5b0 775254 /dev/log
syslogd 25627 root 1w REG 3,3 135744 5652 /var/log/debug
syslogd 25627 root 2w REG 3,3 107459 5651 /var/log/syslog
syslogd 25627 root 3w REG 3,3 107054 58317 /var/log/maillog
syslogd 25627 root 4w REG 3,3 4735 16 /var/log/authlog
Youcanalsospecifyspecialdevicesonthecommandline.Forexample,let’s
see what the user on pts/0 is up to:
rob@mouse:~# lsof /dev/pts/0
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 29816 rob 0u CHR 136,0 2 /dev/pts/0
bash 29816 rob 1u CHR 136,0 2 /dev/pts/0
bash 29816 rob 2u CHR 136,0 2 /dev/pts/0
bash 29816 rob 255u CHR 136,0 2 /dev/pts/0
lsof 30882 root 0u CHR 136,0 2 /dev/pts/0
lsof 30882 root 1u CHR 136,0 2 /dev/pts/0
lsof 30882 root 2u CHR 136,0 2 /dev/pts/0118 | Monitoring
#57 Checking On Open Files and Sockets with lsof
HACK
Ifyouneedtospecifymultipleswitches,theyareORedwitheachotherby
default.Torequireallswitches(thatis,toANDthem)includethe -a flagon
eachswitchyouwanttoAND.Forexample,toseealloftheopenfilesasso-
ciated with vi processes that rob is running, try this:
root@mouse:~# lsof -u rob -ac vi
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
vi 31059 rob cwd DIR 3,3 2824 39681 /home/rob
vi 31059 rob rtd DIR 3,3 408 2 /
vi 31059 rob txt REG 3,3 554504 9799 /usr/bin/vim
vi 31059 rob mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
vi 31059 rob mem REG 3,3 282178 2825 /lib/libncurses.so.5.2
vi 31059 rob mem REG 3,3 76023 2831 /usr/lib/libgpm.so.1.18.0
vi 31059 rob mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
vi 31059 rob mem REG 3,3 249120 11721 /lib/libnss_compat-2.2.3.so
vi 31059 rob mem REG 3,3 357644 11725 /lib/libnsl-2.2.3.so
vi 31059 rob 0u CHR 136,1 3 /dev/pts/1
vi 31059 rob 1u CHR 136,1 3 /dev/pts/1
vi 31059 rob 2u CHR 136,1 3 /dev/pts/1
vi 31059 rob 3u REG 3,3 4096 15 /home/rob/.sushi.c.swp
Ifyou’dliketoexamineopensocketsandtheirassociatedprocesses(likea
netstat -p), try the -i switch:
rob@mouse:~# lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 69 root 3u IPv4 61 TCP *:ssh (LISTEN)
mysqld 126 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
mysqld 128 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
mysqld 129 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
httpd 24905 root 22u IPv4 852520 TCP *:443 (LISTEN)
httpd 24905 root 23u IPv4 852521 TCP *:www (LISTEN)
httpd 28383 www 4u IPv4 917713 TCP nocat.net:www->65.192.187.158:8648
(ESTABLISHED)
httpd 28389 www 4u IPv4 917714 TCP nocat.net:www->65.192.187.158:9832
(ESTABLISHED)
httpd 28389 www 22u IPv4 852520 TCP *:443 (LISTEN)
httpd 28389 www 23u IPv4 852521 TCP *:www (LISTEN)
exim 29879 exim 0u IPv4 557513 TCP *:smtp (LISTEN)
inetd 30563 root 4u IPv4 847222 TCP *:telnet (LISTEN)
inetd 30563 root 5u IPv4 560439 TCP *:cvspserver (LISTEN)
sshd 30973 root 4u IPv4 901571 TCP nocat.net:ssh->some.where.net:52543
(ESTABLISHED)
sshd 30991 rob 4u IPv4 901577 TCP nocat.net:ssh->some.where.else.net:52544
(ESTABLISHED)
Notethatyoumustberoottorun lsof formanyfunctions,includingretriev-
ingopensocketinformation. lsof isacomplexandveryflexibletool,giving
youasmuch(oraslittle)detailasyouneedaboutwhatfilesareinuseby
every running process on your system.Monitor System Resources with top#58
Monitoring | 119
HACK
See also:
•Thelatestversionof lsof canbedownloadedat ftp://vic.cc.purdue.edu/
pub/tools/unix/lsof/.
HACK
#58
Monitor System Resources with top Hack #58
Use the top utility to get a better overview of what your system is up to
The top commandcangiveyouup-to-the-secondreportingofsystemload,
memoryusage,andCPUutilization.Itisdistributedaspartofthe procps
package.Thesimplestwaytogetstartedistosimplyrun top fromthecom-
mand line:
$ top
You’llbepresentedwithascreenfulofinformationupdatedeverytwo
seconds.
3:54pm up 1 day, 16 min, 2 users, load average: 0.00, 0.00, 0.00
38 processes: 37 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 0.7% system, 0.0% nice, 99.2% idle
Mem: 189984K av, 155868K used, 34116K free, 0K shrd, 42444K buff
Swap: 257032K av, 0K used, 257032K free 60028K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
6195 rob 14 0 1004 1004 800 R 0.5 0.5 0:00 top
1 root 8 0 212 212 180 S 0.0 0.1 0:13 init
2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd
4 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd
5 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
6 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
8 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
176 root 9 0 788 788 680 S 0.0 0.4 0:00 syslogd
179 root 9 0 1228 1228 444 S 0.0 0.6 0:00 klogd
182 root 8 0 1228 1228 1104 S 0.0 0.6 0:06 sshd
184 root 8 0 616 616 520 S 0.0 0.3 0:00 crond
186 daemon 9 0 652 652 560 S 0.0 0.3 0:00 atd
197 root 9 0 2544 2544 2396 S 0.0 1.3 0:00 httpd
200 root 9 0 3740 3740 1956 S 0.0 1.9 0:00 named
202 root 9 0 1004 1004 828 S 0.0 0.5 0:00 dhcpd
203 root 9 0 504 504 444 S 0.0 0.2 0:00 agetty
Hit?whiletopisrunningtogetalistofavailablecommands.Acoupleof
veryusefuldisplaykeysareM(whichsortsonresidentmemorysize),P
(whichsortsbyCPUusageagain),S(totogglecumulativeruntime,thatis,
howlongeachprocessandallofitschildrenhavebeenrunning,inCPUsec-
onds), and i (to stop displaying idle processes.)
Ifyou’rerunning top asroot,thereareacoupleofotherinteractiveandsort-
ingcommandsthatyou’lllikelyfinduseful.Theukeyletsyoufilteroutall
Download from Wow! eBook <www.wowebook.com>120 | Monitoring
#59 Constant Load Average Display in the Titlebar
HACK
processesexceptthoseownedbyagivenuser.Followthatupwithk,which
letsyouinteractivelykillagivenPID(withanysignalyoulike.)Thiscanbe
reallyhandyforhuntingdownrunawayprocesses,andkillingthemfrom
insidetop(maybeevencopying-and-pastingtheoffendingPIDtoavoidthe
dreaded kill typo).
Itcanbehandytoleave top runninginaterminalwindowonbusymachines
thatyou’reloggedinto,butaren’totherwiseworkingon.Ifyou’dliketosee
acontinualloadaveragedisplayinthetitlebarofaloginwindow(forasort
ofmini-topwhileremaininginyourshell),checkout“ConstantLoadAver-
age Display in the Titlebar” [Hack #59].
See also:
•“Manipulating Processes Symbolically withprocps” [Hack #17]
•“Constant Load Average Display in the Titlebar” [Hack #59]
• procps home: ftp://people.redhat.com/johnsonm/procps/
HACK
#59
Constant Load Average Display in the
Titlebar Hack #59
Make your title bar work harder, so you don’t have to.
IfyouhavemanagedaLinuxserverforanylengthoftime,you’reprobably
intimatelyfamiliarwiththe top utility.(Ifnot,stopreadingandimmediately
gotypetopinthenearestterminalwindow,andhit?whenyougetbored.)
Ifyou’vemanagedanumberofLinuxservers,thenyouprobablyknowwhat
it’sliketohaveseveral topsrunninginmultiplewindows,allcompetingfor
desktop real estate.
Incomputing,wastedresourcesareresourcesthatcouldbebetterspent
helpingyou.Whynotrunaprocessthatupdatesthetitlebarofyourtermi-
nalwiththecurrentloadaverageinrealtime,regardlessofwhatelseyou’re
running?
Save this as a script called tl, and save it to your ~/bin directory:
Listing: tl
#!/usr/bin/perl -w
use strict;
$|++;
my $host=`/bin/hostname`;
chomp $host;
while(1) {Network Monitoring with ngrep#60
Monitoring | 121
HACK
open(LOAD,"/proc/loadavg") || die "Couldn't open /proc/loadavg: $!\n";
my @load=split(/ /,<LOAD>);
close(LOAD);
print "\033]0;";
print "$host: $load[0] $load[1] $load[2] at ", scalar(localtime);
print "\007";
sleep 2;
}
Whenyou’dliketohaveyourtitlebarreplacedwiththename,loadaverage,
andcurrenttimeofthemachineyou’reloggedinto,justrun tl&.Itwillhap-
pilygoonrunninginthebackground,evenifyou’rerunninganinteractive
programlike vim.Ifyouhaveyourtitlebaralreadysettoshowthecurrent
workingdirectory,noproblem.Whenyou cd,thecurrentworkingdirectory
isflashedmomentarily,andthenreplacedwiththetimeandloadaverage
again.Needtoseethatdirectoryonemoretime?JusthitEnteronablank
line, and it will flash again.
Nowinsteadofspreadingterminalsaroundandcompletelycoveringyour
desktop,youcanstackapileofthemtogether(leavingonlyyourtitlebars
showing),andenjoyan“ataglance”viewofhowhardeachmachineiswork-
ing.Whenyouneedtoworkonamachine,justbringthatwindowtothefore-
ground,andimmediatelybeginworking.Ofcourse,thisisallavailabletoyou
without installing a single piece of software on your local machine.
Whenyou’refinished,don’tforgetto killalltl beforeloggingout.Orif
you’re terminally lazy, try this:
$ echo 'killall tl > /dev/null 2>&1' >> ~/.bash_logout
Thatwillkillallofyourrunning tl jobsonlogout,withoutevenhavingtolift
a finger. System administration just keeps getting easier, doesn’t it?
HACK
#60
Network Monitoring with ngrep Hack #60
See who’s doing what, with a grep for your network interface
The ngrep utilityisaninterestingpacketcapturetool,similarto tcpdump or
snoop.Itisuniqueinthatitattemptstomakeitaseasyaspossibletomatch
whichcapturedpacketstoprint,byusinga grep compatibleformat(com-
pletewithregularexpressionsandabunchofGNUgrep’sswitches).Italso
converts the packets to ASCII (or hex) before printing.
Forexample,toseethecontentsofallhttpGETrequeststhatpassthrough
your router, try this:
# ngrep -q GET122 | Monitoring
#60 Network Monitoring with ngrep
HACK
Ifyou’reonlyinterestedinaparticularhost,protocol,orport(orother
packetmatchingcriteria),youcanspecifyabpffilteraswellasadatapat-
tern. It uses a syntax similar to tcpdump:
# ngrep -qi rob@nocat.net port 25
T 10.42.4.7:65174 -> 209.204.146.26:25 [AP]
RCPT TO:<rob@nocat.net>..
T 209.204.146.26:25 -> 10.42.4.7:65174 [AP]
250 2.1.5 <rob@nocat.net>... Recipient ok..
T 10.42.4.7:65174 -> 209.204.146.26:25 [AP]
Date: Sun, 8 Sep 2002 23:55:18 -0700..Mime-Version: 1.0 (Apple Message fram
ework v543)..Content-Type: text/plain; charset=US-ASCII; format=flowed..Sub
ject: Greetings.....From: John Doe <johnd@somewhere.else.com>..To: rob@noca
t.net..Content-Transfer-Encoding: 7bit..Message-Id: <19DB8C16-C3C1-11D6-B23
9-0003936D6AE0@somewhere.else.com>..X-Mailer: Apple Mail v2)....What does t
hat pgp command you mentioned do again?....Thanks,....--A Friend....
Since ngrep printstoSTDOUT,youcandopost-processingontheoutputto
makeaniceprintingfilter.Ifyouprocesstheoutputyourself,addthe -l
switchtomaketheoutputlinebuffered.Forexample,ifyou’reinterestedin
whatpeopleonyournetworkaresearchingforonline,trysomethinglike
this bit of Perl.
Listing: go-ogle
#!/usr/bin/perl
use Socket;
$|++;
open(NG,"ngrep -lqi '(GET|POST).*/(search|find)' |");
print "Go ogle online.\n";
my ($go,$i) = 0;
my %host = ( );
while(<NG>) {
if(/^T (\d+\.\d+.\d+\.\d+):\d+ -> (\d+\.\d+\.\d+\.\d+):80/) {
$i = inet_aton($1);
$host{$1} ||= gethostbyaddr($i, AF_INET) || $1;
$i = inet_aton($2);
$host{$2} ||= gethostbyaddr($i, AF_INET) || $2;
print "$host{$1} -> $host{$2} : ";
$go = 1;
next;
}
if(/(q|p|query|for)=(.*)?(&|HTTP)/) {
next unless $go;
my $q = $2;
$q =~ s/(\+|&.*)/ /g;
$q =~ s/%(\w+)/chr(hex($1))/ge;Scanning Your Own Machines with nmap#61
Monitoring | 123
HACK
print "$q\n";
$go = 0;
}
}
Icallthescript go-ogle.Thiswillrunan ngrep lookingforanyGETorPOST
requestthatincludes search or find somewhereintheURL.Theresultslook
something like this:
Go ogle online.
caligula.nocat.net -> www.google.com : o'reilly mac os x conference
caligula.nocat.net -> s1.search.vip.scd.yahoo.com : junk mail $$$
tiberius.nocat.net -> altavista.com : babel fish
caligula.nocat.net -> 166-140.amazon.com : Brazil
livia.nocat.net -> 66.161.12.119 : lart
Itwillunescapeencodedstringsinthequery(notethe ‘ inthegooglequery
andthe$$$fromyahoo).ItwillalsoconvertIPaddressestohostnamesfor
you(since ngrep doesn’tseemtohavethatfeature,probablysoitcanopti-
mizecapturingforspeed).Thelasttworesultsareinteresting:theBrazilquery
wasactuallyrunon http://www.imdb.com/,andthelastonewasto http://www.
dictionary.com/.EvidentlyIMDBisnowinapartnershipwithAmazon,and
Dictionary.com’ssearchmachinedoesn’thaveaPTRrecord.It’samazing
how much you can learn about the world by watching other people’s packets.
Notethatyoumustberoottorun ngrep,andforbestresults,itshouldbe
run from the router at the edge of your network.
See also:
•man ngrep
• http://www.packetfactory.net/Projects/ngrep/
HACK
#61
Scanning Your Own Machines with nmap Hack #61
Find out when servers and services come online anywhere on your network
Ifyouhaven’tuseditbefore,nmapisatremendouslyusefultoolforidenti-
fyingmachinesandservicesonyournetwork.Itwillperformanumberof
differenttypesofnetworkscanning(fromstandardTCPandUDPtomore
exoticscanslikestealthTCPSYNscans,XmasTreeandNULLprobes,and
a bunch of other fun options).
EvenmoreinterestingistheOSfingerprintingcode,whichanalyzespackets
returnedbythetargetmachineandcomparestheresultsagainstadatabase
ofknownoperatingsystems.Thisisafascinatingbitofcode,inthatitcan
typicallyidentifytheremoteside’soperatingsystemwithoutconnectingto
anyactualservices,andevenreturnanestimateduptimeforthemachine
being scanned.124 | Monitoring
#61 Scanning Your Own Machines with nmap
HACK
ToperformastandardportsweepwithOSfingerprinting,trythe -O switch:
rob@catlin:~# nmap -O caligula
Starting nmap V. 3.00 ( www.insecure.org/nmap/ )
Interesting ports on caligula.rob.nocat (10.42.4.7):
(The 1600 ports scanned but not shown below are in state: closed)
Port State Service
22/tcp open ssh
Remote operating system guess: Mac OS X 10.1 - 10.1.4
Uptime 5.760 days (since Tue Sep 3 19:14:36 2002)
Nmap run completed -- 1 IP address (1 host up) scanned in 31 seconds
Ifyou’dliketo nmap yourentirenetworkandhaveabitoftimetokill,you
canspecifyanetworkandsubnetonthecommandline.ThisperformsaTCP
SYN scan and fingerprinting for the first 64 addresses of 10.42.4.0:
root@catlin:~# nmap -OsS 10.42.4.0/26
Since nmap printstoSTDOUT,youcansavetheoutputofascanrunand
compareitagainstpreviousrunsforadifferentialreport,quiteeasily.We’ll
runanXmastreescanand grep outacoupleoflines(liketheruntime)to
eliminate false positives:
root@catlin:~# nmap -sX 10.42.4.0/26 | egrep -v '^(Nmap|Starting)' \
> nmap.output
Let’s run the same command again (say, the next day, at a random hour):
root@catlin:~# nmap -sX localhost | egrep -v '^(Nmap|Starting)' \
> nmap.output2
and let’s do a context diff to see what changed:
root@catlin:~# diff -c nmap.output*
*** nmap.output Mon Sep 9 14:45:06 2002
--- nmap.output2 Mon Sep 9 14:45:21 2002
***************
*** 1,7 ****
Interesting ports on catlin.rob.nocat (10.42.4.6):
! (The 1598 ports scanned but not shown below are in state: closed)
Port State Service
22/tcp open ssh
53/tcp open domain
80/tcp open http
--- 1,8 ----
Interesting ports on catlin.rob.nocat (10.42.4.6):
! (The 1597 ports scanned but not shown below are in state: closed)
Port State Service
+ 21/tcp open ftp
22/tcp open sshDisk Age Analysis#62
Monitoring | 125
HACK
53/tcp open domain
80/tcp open http
root@catlin:~#
Fascinating.Itlookslikecatlinhaspickedupan ftp serveratsomepoint.This
techniquewillfindnew(anddead)hostsandserviceseachtimeitisrun.By
keepinganarchiveof nmap output(perhapsloggedtotimeanddateencoded
files,oreventoadatabase)youcankeepalogofthestateofallmachineson
yournetwork.Turningitintoashellscriptandrunningitfrom cron isleftas
anexercise(ahopefullyfun,anddefinitelyworthwhileexercise)forthereader.
See also:
•nmap’s home: http://www.insecure.org/nmap/
HACK
#62
Disk Age Analysis Hack #62
Easily identify which parts of your disk change frequently
Howcanyouquicklytellwhichpartsofyourfilesystemaremodifiedfre-
quentlyandwhichhaven’tchangedinmonths?It’sverystraightforward,
with a proper application of Perl.
HereisaPerlscriptthatwillperformdiskaginganalysisonafilesystem.The
programbreaksdowndiskspacetwoways,bylastmodifieddateandlast
accessed date. A sample run looks like:
% diskage /usr/local
Disk aging analysis for /usr/local:
last num last num
Age (days) modified files accessed files
0 - 30 260403 Kb 817 140303 Kb 6968
31 - 60 11789 Kb 226 23140 Kb 199
61 - 90 40168 Kb 1126 1087585 Kb 31625
91 - 180 118927 Kb 995 0 Kb 0
181 - 365 85005 Kb 1889 0 Kb 0
366 - 9999 734735 Kb 33739 0 Kb 0
----------- ----- ----------- -----
Total 1251029 Kb 38792 1251029 Kb 38792
Youcanrunthescriptwiththe -v optiontolistthelastmodifiedandlast
accessed days for every file in the filesystem.
Listing: diskage
#!/usr/local/bin/perl
#
# Disk aging report generator126 | Monitoring
#62 Disk Age Analysis
HACK
# Written by Seann Herdejurgen
#
# May 1998
use File::Find;
# Initialize variables
@levels=(30,60,90,180,365,9999);
# Check for verbose flag
if ($ARGV[0] eq "-v") {
$verbose++;
shift(@ARGV);
}
$ARGV[0]=$ENV{'PWD'} if ($ARGV[0] eq "");
foreach $dir (@ARGV) {
foreach $level (@levels) {
$modified{$level}=0;
$accessed{$level}=0;
$mfiles{$level}=0;
$afiles{$level}=0;
}
print("\nDisk aging analysis for $dir:\n\n");
print (" mod acc size file\n") if ($verbose);
# Traverse desired filesystems
find(\&wanted,$dir);
print(" last num last num\n");
print(" Age (days) modified files accessed files\n");
$msize=$asize=$mtotal=$atotal=$lastlevel=0;
foreach $level (@levels) {
printf("%4d - %4d %8d Kb %5d %8d Kb %5d\
n",$lastlevel,$level,$modified{$level}/
1024,$mfiles{$level},$accessed{$level}/1024,$afiles{$level});
$msize+=$modified{$level}/1024;
$asize+=$accessed{$level}/1024;
$mtotal+=$mfiles{$level};
$atotal+=$afiles{$level};
$lastlevel=$level+1;
}
printf(" ----------- ----- ----------- -----\n");
printf(" Total %8d Kb %5d %8d Kb %5d\n",$msize,$mtotal,$asize,$atotal);
}
exit;
sub wanted {
(($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size) = lstat($_));
$mod=int(-M _);Cheap IP Takeover#63
Monitoring | 127
HACK
$acc=int(-A _);
foreach $level (@levels) {
if ($mod<=$level) { $modified{$level}+=$size; $mfiles{$level}++; last; }
}
foreach $level (@levels) {
if ($acc<=$level) { $accessed{$level}+=$size; $afiles{$level}++; last; }
}
printf("%4d %4d %6d %s\n",$mod,$acc,$size,$_) if ($verbose);
}
HACK
#63
Cheap IP Takeover Hack #63
Accomplish IP takeover with ping, bash, and a simple network utility
Directingtraffictooneofseveralmachinesisfairlystraightforwardwhen
usinground-robinDNS,asdiscussedin“DistributingServerLoadwith
Round-RobinDNS” [Hack#79].Butwhathappenswhenoneofthoseservers
becomesunavailable?Here’soneschemeformonitoringthehealthof
another server and standing in for it if it fails.
First,weneedtomakeadistinctionbetweentheserver’s“real”IPaddress,
andtheIP(orIPs)thatitactuallyservespubliccontentfrom.Forthisexam-
ple,we’llbereferringtotwoservers,PinkyandBrain.PinkyusestheIP
address208.201.239.12forits“real”IPoneth0,andalsohasanIPaliasof
208.201.239.36oneth0:0.Brainuses208.201.239.13oneth0,and208.201.
239.37oneth0:0.Ifyou’veneverusedIPaliasesbefore,here’sthevery
quick HOWTO:
# ifconfig eth0:0 1.2.3.4
Voila,youhaveanotherIPaddress(1.2.3.4)boundtoeth0,calledeth0:0.
YouusedtohavetospecificallycompileIPaliasingintothekernel,butthis
optionseemstohavegoneawayinrecentkernelsandisapparentlyonby
default.OneimportantthingtorememberaboutIPaliasesisthatifthe
interfacetowhichitisbound(inthiscase,eth0)iseverbroughtdown,then
allofitsassociatedaliasesarealsobroughtdown.Youcanalsomakethe
aliasanyalphanumericstring,althoughsomeversionsof ifconfig onlydis-
play the first four or five characters of the alias when displaying interfaces.
OncePinkyandBrainhavetheirrespectiveeth0:0set,bindaservice(like
Apache)totheiraliasedIPs,andsetupround-robinDNStopointtoboth
withasinglehostname(see“DistributingServerLoadwithRound-Robin
DNS” [Hack#79]).We’llassumethatwe’resettingupredundantwebservicefor
www.oreillynet.com, resolving to either 208.201.239.36 or 208.201.239.37.
Nowthatroughlyhalfofthetrafficisgoingtoeachserver,we’llneedPinky
andBraintomonitorthehealthofeachother.Thiscanbedonebypinging
eachother’srealIPaddress,andwatchingtheresults.Savethefollowing
into a script, and install it on Pinky.128 | Monitoring
#63 Cheap IP Takeover
HACK
Listing: takeover
#!/bin/bash
OTHER="brain"
PUBLIC="208.201.239.37"
PAUSE=3
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin
MISSED=0
while true; do
if ! ping -c 1 -w 1 $OTHER > /dev/null; then
((MISSED++))
else
if [ $MISSED -gt 2 ]; then
ifconfig eth0:$OTHER down
fi
MISSED=0
fi;
if [ $MISSED -eq 2 ]; then
ifconfig eth0:$OTHER $PUBLIC
#
# ...but see discussion below...
#
fi
sleep $PAUSE;
done
Naturally,setOTHERto“pinky”andPUBLICto“208.201.239.36”onthe
copy that runs on Brain.
Let’ssupposethatBrainsuddenlystopsrespondingon208.201.239.13(saya
networktechaccidentallypulledthewrongplugwhenworkingontherack).
Aftermissing3pingsinarow,Pinkywillleapintoaction,bringingupeth0:
brainupas208.201.239.37,thepublicIPthatBrainissupposedtobeserving.
ItwillthencontinuetowatchBrain’srealIPaddress,andrelinquishcontrol
whenitisbackonline.The ping-c1-w1 means“sendonepingpacket,and
timeoutafteronesecond,nomatterwhathappens.”pingwillreturnnon-zero
if the packet didn’t come back in the one second time limit.
Butthisisn’tquitetheentiresolution.AlthoughPinkyisnowansweringfor
Brain,anymachinesonthesamenetworkasthetwoservers(notably,the
routerjustupstreamatyourISP)willhavethewrongMACaddresscachedfor
208.201.239.37.WiththewrongMACaddresscached,notrafficwillflowto
Pinky,sinceitwillonlyrespondtopacketsthatbearitsownMACaddress.
Howcanwetellallofthemachinesonthe208.201.239.0networkthatthe
MAC address for 208.201.239.37 has been updated?Running ntop for Real-Time Network Stats#64
Monitoring | 129
HACK
Onewayistousethe send_arp utilityfromtheHighAvailabilityLinux
project.Thisveryhandy(andtiny)utilitywillcraftanARPpackettoyour
specificationsandsendittoaMACaddressofyourchoiceonthelocalnet-
work.Ifwespecifyallones(i.e., ff:ff:ff:ff:ff:ff)forthedestination,
theniteffectivelybecomesabroadcastARPpacket.Whilemostrouters
won’tupdatetheirARPtableswhentheyseeunrequestedARPbroadcasts,
suchapacketwillsignalthemtoresendanARPrequest,towhichPinkywill
obliginglyreply.Theadvantageofusingbroadcastisthatitwillsignalall
machinesonthesubnetsimultaneously,insteadofhavingtotrackallofthe
MAC addresses of machines that need updating.
Thesyntaxof send_arp is send_arp[SourceIP][SourceMAC][TargetIP]
[TargetMAC].Forexample,oursimplemonitoringscriptaboveshouldrun
the following when it detects that Brain is down:
send_arp 208.201.239.37 00:11:22:aa:bb:cc 208.201.239.37 fffffffffff
(Where 00:11:22:aa:bb:cc isthehardwareMACaddressofPinky’seth0.)
ThescriptcancontinuetowatchtowatchwhenBrain’srealIPaddress
(208.201.239.13)becomesavailable.Whenitdoes,wecanbringeth0:brain
backdownandletBrainworryaboutupdatingtheARPcacheagain(which
it should be set to do on boot).
Thereareanumberofimprovementsthatcouldbemadetothistechnique.
Foronething,justbecause208.201.239.13isupdoesn’tguaranteethat208.
201.239.37isalsoavailable.Also,pingisn’tthebesttestforserviceavail-
ability(abettertestmightbetoactuallyrequestawebpagefromtheother
machine and make sure that it has a closing </html> tag).
Theseimprovementsareleftasanexercisetoyou,dearreader.Everysiteisdif-
ferent,soyou’llneedtofindthetechniquethatworksbestwiththetoolsthat
you have at hand. After all, that’s exactly what a hack is, isn’t it?
See also:
•The Fake packagefromHighAvailabilityLinux, http://www.linux-ha.org/
failover/
•“Distributing Server Load with Round-Robin DNS” [Hack #79]
HACK
#64
Running ntop for Real-Time Network Stats Hack #64
See who’s doing what on your network over time with ntop
Ifyou’relookingforreal-timenetworkstatistics,youshouldcheckoutthe
terrific ntop tool.Itisafull-featuredprotocolanalyzerwithawebfront-end,
completewithSSLandGDgraphingsupport. ntop isn’tlightweight(requir-
ingmoreresourcesdependingonthesizeofyournetworkandthevolumeof130 | Monitoring
#64 Running ntop for Real-Time Network Stats
HACK
nettraffic)butcangiveyouaveryniceoverview(andsomecriticaldetails)
about who’s talking to whom on your network.
ntop needstoinitiallyrunasroot(tothrowyourinterfacesintopromiscuous
modeandstartcapturingpackets),butthenreleasesitsprivilegestoauser
thatyouspecify.Ifyoudecidetorun ntop forlongperiodsoftime,you’ll
probablybehappiestrunningitonadedicatedmonitoringbox(withfew
other services running on it for security and performance reasons.)
Here’saquickreferenceonhowtoget ntop upandrunningquickly.First,
create an ntop user and group:
root@gemini:~# groupadd ntop
root@gemini:~# useradd -c "ntop user" -d /usr/local/etc/ntop \
-s /bin/true -g ntop ntop
Thenunpackandbuild ntop aspertheinstructionsin docs/BUILD-NTOP.
txt.We’llassumethatyouhavethesourcetreeunpackedin /usr/local/src/
ntop-2.1.3/.
Create a directory for ntop in which to keep its capture database:
root@gemini:~# mkdir /usr/local/etc/ntop
(Note that it should be owned by root, and not by the ntop user.)
Ifyou’dliketouseSSLforhttps(insteadofstandardhttp),thencopythe
default SSL key to /usr/local/etc/ntop:
root@gemini:# cp /usr/local/src/ntop-2.1.3/ntop/*pem /usr/local/etc/ntop
NotethatthedefaultSSLkeywillnotbebuiltwiththecorrecthostnamefor
yourserver.Ifyou’dliketomakeyourown(andeliminatetheSSLwarningsin
yourbrowser),checkout“GeneratinganSSLcertandCertificateSigning
Request” [Hack#93] and“CreatingYourOwnCA” [Hack#94] foranexampleof
how to make your own key and sign it with your own CA.
Now we initialize the ntop databases and set an administrative password:
root@gemini:~# ntop -A -u ntop -P /usr/local/etc/ntop
21/Sep/2002 20:30:23 Initializing GDBM...
21/Sep/2002 20:30:23 Started thread (1026) for network packet analyser.
21/Sep/2002 20:30:23 Started thread (2051) for idle hosts detection.
21/Sep/2002 20:30:23 Started thread (3076) for DNS address resolution.
21/Sep/2002 20:30:23 Started thread (4101) for address purge.
Please enter the password for the admin user:
Please enter the password again:
21/Sep/2002 20:30:29 Admin user password has been set.
Finally,run ntop asadaemon,andstarttheSSLserveronyourfavoriteport
(4242, for example):
root@gemini:~# ntop -u ntop -P /usr/local/etc/ntop -W4242 -d
Download from Wow! eBook <www.wowebook.com>Running ntop for Real-Time Network Stats#64
Monitoring | 131
HACK
Bydefault, ntop willalsorunastandardhttpserveronport3000.Youshould
stronglyconsiderlockingdownaccesstotheseportsatyourfirewall,orby
usingcommandlineiptablesrules,asin“CreatingaFirewallfromtheCom-
mandLineofanyServer” [Hack#45].Let ntop runforawhile,thenconnectto
https://your.server.here:4242/.Youcanfindoutallsortsofdetailsaboutwhat
traffic has been seen on your network, as inFigure5-1.
Andevenlookatvariousstatisticsgraphically,asaneasy-to-read(and
pretty-to-print) pie chart, as inFigure5-2.
Whiletoolslike tcpdump and ethereal willgiveyoudetailed,interactiveanalysis
ofnetworktraffic, ntop deliversawealthofstatisticalinformationinaslickand
easy-to-usewebinterface.Whenproperlyinstalledandlockeddown,itmay
well become a favorite tool in your network analysis tool chest.
See also:
• http://www.ntop.org/
• http://www-serra.unipi.it/~ntop/ntop.html
Figure5-1.ntop will generate real-time statistical graphs of your network traffic132 | Monitoring
#65 Monitoring Web Traffic in Real Time with httptop
HACK
HACK
#65
Monitoring Web Trafﬁc in Real Time with
httptop Hack #65
See who’s hitting your web server the hardest up to the second with httptop
Advancedhttploganalysispackageslike analog cangiveyouvery
detailedreportonwhohasbeenlookingatwhatonyoursite.Unfortu-
nately,logprocessingjobscantakequitealongtimetorunonabusy
site,givingyoudetailedretrospectivereportingbutnothingaboutwhat’s
happening right now.
Anotherapproachistologallrelevantwebactivitytoadatabaseandper-
formqueriesonitforreal-timestatistics.Thismethodcangiveyouinstanta-
neous,to-the-hitreporting,aslongasyouhaveadatabaseandwebserver
powerhousecapableofsupportingtheadditionalload.Onaverybusysite,
this simply isn’t practical.
Somewhereinbetweenlies httptop,aPerlscriptthatattemptstogive top-like
reportinginrealtime,withouttheoverheadofadatabaseserver.Itexpects
to be given the path to an access_log file. Run the command like this:
$ httptop -f combined /usr/local/apache/logs/access_log
Figure5-2.When run overtime, ntop will reveal great detail about how individual clients
use your networkMonitoring Web Traffic in Real Time with httptop#65
Monitoring | 133
HACK
You’llbepresentedwithascreenthatrefresheseverycoupleofseconds,
sortedonhitspersecond.Justaswith top,hit ? foralistingofavailable
keys.Theyincludesuchnicetiesassortingbymostrecenthitsortotalhits,
andoptionallydisplayingthereferringURLordomain,therequestedURI,
or remote requesting host.
IfyouhaveabunchofVirtualHostsandwouldliketobeabletosimulta-
neously run httptop across all of them, try this:
Add a line like this to each of your VirtualHost entries:
CustomLog /usr/local/apache/logs/combined-log vhost
andthefollowinglogdefinition(whichshouldappearallononeline)some-
where in your global configuration:
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
vhost
Nowyoucanwatchactivityonallofyourhostssimultaneouslywithacom-
mand, as:
$ httptop -f vhost /usr/local/apache/logs/combined-log
Notethat httptop requirestheTime::HiResandFile::TailPerlmodulestobe
installedfirst.Ifyoudon’tfeelliketypingallofthisin, httptop (andmostof
theexamplesinthisbook)areavailablefromthe examples.oreilly.com web
site. See the introduction for the exact URL.
Listing: httptop
#!/usr/bin/perl -w
#
=head1 NAME
httptop - display top(1)-like per-client HTTP access stats
=head1 SYNOPSIS
httptop [-f <format>] [-r <refresh_secs>] [-b <backtrack_lines>] <logdir |
path_to_log>
=cut
use Time::HiRes qw( time );
use File::Tail ();
use Term::ReadKey;
use Getopt::Std;
use strict;134 | Monitoring
#65 Monitoring Web Traffic in Real Time with httptop
HACK
### Defaults you might be interested in adjusting.
my $Update = 2; # update every n secs
my $Backtrack = 250; # backtrack n lines on startup
my @Paths = qw(
%
/title/%/logs/access_log
/var/log/httpd/%/access_log
/usr/local/apache/logs/%/access_log
);
my $Log_Format = "combined";
my %Log_Fields = (
combined => [qw/ Host x x Time URI Response x Referer Client /],
vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
);
### Constants & other thingies. Nothing to see here. Move along.
my $Version = "0.4.1";
sub by_hits_per () { $b->{Rate} <=> $a->{Rate} }
sub by_total () { $b->{Total} <=> $a->{Total} }
sub by_age () { $a->{Last} <=> $b->{Last} }
my $last_field = "Client";
my $index = "Host";
my $show_help = 0;
my $order = \&by_hits_per;
my $Help = "htlwufd?q";
my %Keys = (
h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
w => [ "Show remote host" => sub { $index = "Host" } ],
u => [ "Show requested URI" => sub { $index = "URI" } ],
f => [ "Show referring URL" => sub { $index = "Referer" } ],
d => [ "Show referring domain" => sub { $index = "Domain" } ],
'?' => [ "Help (this thing here)" => sub { $show_help++ } ],
q => [ "Quit" => sub { exit } ]
);
my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
my @Record_Fields = qw/ Host URI Referer Domain /;
my $Max_Index_Width = 50;
my $Initial_TTL = 50;
my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
my %Term = (
HOME => "\033[H",
CLS => "\033[2J",
START_TITLE => "\033]0;", # for xterms etc.
END_TITLE => "\007",Monitoring Web Traffic in Real Time with httptop#65
Monitoring | 135
HACK
START_RV => "\033[7m",
END_RV => "\033[m"
);
my ( %hist, %opt, $spec );
$SIG{INT} = sub { exit };
END { ReadMode 0 };
### Subs.
sub refresh_output
{
my ( $cols, $rows ) = GetTerminalSize;
my $show = $rows - 3;
my $count = $show;
my $now = (shift || time);
for my $type ( values %hist ) {
for my $peer ( values %$type ) {
# if ( --$peer->{_Ttl} > 0 ) {
my $delta = $now - $peer->{Start};
if ( $delta >= 1 ) {
$peer->{ Rate } = $peer->{ Total } / $delta;
} else {
$peer->{ Rate } = 0
}
$peer->{ Last } = int( $now - $peer->{ Date } );
# } else {
# delete $type->{$peer}
# }
}
}
$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values
%{$hist{$index}};
my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
my $first = 0;
$first = ( $first <= $_ ? $_ + 1 : $first ) for map { $_ ? length($_->
{$index}) : 0 } @list;
$first = $Max_Index_Width if $Max_Index_Width < $first;
print $Term{START_TITLE}, "Monitoring $spec at: ", scalar localtime,
$Term{END_TITLE} if $ENV{TERM} eq "xterm"; # UGLY!!!
my $help = "Help/?";
my $head = sprintf( "%-${first}s %6s %4s %4s %s (%d total)",
$index, qw{ Hits/s Tot Last }, $last_field,
scalar keys %{$hist{$index}}
);136 | Monitoring
#65 Monitoring Web Traffic in Real Time with httptop
HACK
#
# Truncate status line if need be
#
$head = substr($head, 0, ($cols - length($help)));
print @Term{"HOME", "START_RV"}, $head, " " x ($cols - length($head) -
length($help)), $help, $Term{END_RV}, "\n";
for ( @list ) {
# $_->{_Ttl}++;
my $line = sprintf( "%-${first}s %6.3f %4d %3d %s",
substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last },
$last_field)} );
if ( length($line) > $cols ) {
substr( $line, $cols - 1 ) = "";
} else {
$line .= " " x ($cols - length($line));
}
print $line, "\n";
}
print " " x $cols, "\n" while $count++ < $show;
}
sub process_line
{
my $line = shift;
my $now = ( shift || time );
my %hit;
chomp $line;
@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /"([^"]+)"|\[([^]]+)\
]|\s/o, $line ) );
$hit{ URI } =~ s/HTTP\/1\S+//gos;
$hit{ Referer } = "<unknown>" if not $hit{Referer} or $hit{Referer} eq "-";
( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
$hit{ Client } ||= "<none>";
$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
$hit{ Client } =~ s/[^\x20-\x7f]//gos;
# if $now is negative, try to guess how old the hit is based on the time
stamp.
if ( $now < 0 ) {
my @hit_t = ( split( m![:/\s]!o, $hit{ Time } ))[ 0 .. 5 ];
my @now_t = ( localtime )[ 3, 4, 5, 2, 1, 0 ];
my @mag = ( 3600, 60, 1 );
# If the hit didn't parse right, or didn't happen today, the hell with it.
return unless $hit_t[2] == ( $now_t[2] + 1900 )
and $hit_t[1] eq $Months[ $now_t[1] ]
and $hit_t[0] == $now_t[0];Monitoring Web Traffic in Real Time with httptop#65
Monitoring | 137
HACK
splice( @hit_t, 0, 3 );
splice( @now_t, 0, 3 );
# Work backward to the UNIX time of the hit.
$now = time;
$now -= (shift( @now_t ) - shift( @hit_t )) * $_ for ( 3600, 60, 1 );
}
$hit{ Date } = $now;
for my $field ( @Record_Fields ) {
my $peer = ( $hist{$field}{$hit{$field}} ||= { Start => $now, _Ttl =>
$Initial_TTL } );
@$peer{ @Display_Fields } = @hit{ @Display_Fields };
$peer->{ Total }++;
}
}
sub display_help {
my $msg = "httptop v.$Version";
print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, "\n\n";
print " " x 4, $_, " " x 8, $Keys{$_}[0], "\n" for ( split "", $Help );
print "\nPress any key to continue.\n";
}
### Init.
getopt( 'frb' => \%opt );
$Backtrack = $opt{b} if $opt{b};
$Update = $opt{r} if $opt{r};
$Log_Format = $opt{f} if $opt{f};
$spec = $ARGV[0];
die <<End unless $spec and $Log_Fields{$Log_Format};
Usage:$0[-f<format>][-r<refresh_secs>][-b<backtrack_lines>]<logdir|
path_to_log>
Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
End
for ( @Paths ) {
last if -r $spec;
( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
}
die "No access_log $ARGV[0] found.\n" unless -r $spec;
my $file = File::Tail->new(
name => $spec,
interval => $Update / 2,
maxinterval => $Update,
tail => $Backtrack,
nowait => 1
) or die "$spec: $!";138 | Monitoring
#65 Monitoring Web Traffic in Real Time with httptop
HACK
my $last_update = time;
my ( $line, $now );
# Backtracking.
while ( $Backtrack-- > 0 ) {
last unless $line = $file->read;
process_line( $line, -1 );
}
$file->nowait( 0 );
ReadMode 4; # Echo off.
print @Term{"HOME", "CLS"}; # Home & clear.
refresh_output;
### Main loop.
while (defined( $line = $file->read )) {
$now = time;
process_line( $line, $now );
while ( $line = lc ReadKey(-1) ) {
$show_help = 0 if $show_help;
$Keys{$line}[1]->() if $Keys{$line};
}
if ( $show_help == 1 ) {
display_help;
$show_help++; # Don't display help again.
} elsif ( $now - $last_update > $Update and not $show_help ) {
$last_update = $now;
refresh_output( $now );
}
}
__END__
=head1 DESCRIPTION
httptop is intended to be a top(1)-equivalent for httpd access activity.
httptop should be invoked with the path to an Apache access_log, or
alternatelyastringthatuniquelyidentifiesthedirectoryinwhichtofind
the access_log. The search paths can be configured in the source.
httptop has limited flexibility for dealing with logs of different formats.
Run 'httptop' without options to see which format names are available.
While httptop is running, you can obtain a list of terminal commands by
pressing '?'. As with top(1), pressing 'q' quits.139
Chapter6 CHAPTER SIX
SSH
Hacks #66–71
OfallthetypesofhacksIencounteredwhenpreparingthisbook,itbecame
obviousearlyonthat ssh wouldrequireitsownchapter.The ssh toolpro-
videsaveryflexibleandcryptographicallysecuremethodforconnecting
datastreamstogetherbetweenmachinesoveranetwork.Sincethecom-
mandlineonaLinuxsystemessentiallyconsistsofreadingdatafrom(and
writingdatato)filesandpipelines, ssh makesitpossibletoslingdataaround
anetworkmoreorlessasifeverythingweretakingplaceonasingle
machine.Itdoesthisinafast,safe,andintuitiveway,andmakesforsome
very interesting (and powerful) hacks.
Thereareacoupleofversionsof ssh availableforLinux.We’llassumethat
you’reusingOpenSSHv3.4p1orlaterintheexamplesforthissection.
OpenSSHshipswithallmajorLinuxdistributions,andisavailableat http://
www.openssh.com/.
HACK
#66
Quick Logins with ssh Client Keys Hack #66
Usingsshkeysinsteadofpasswordauthenticationtospeedupandautomate
logins
Whenyou’reanadminonmorethanafewmachines,beingabletonavi-
gatequicklytoashellonanygivenserveriscritical.Havingtotype“sshmy.
server.com”(followedbyapassword)isnotonlytedious,butitbreaksone’s
concentration.Suddenlyhavingtoshiftfrom“where’stheproblem?”to
“gettingthere”andbackto“what’sallthis,then?”hasledmorethanone
admintoprematuresenility.Itpromotesthedigitalequivalentof“whydidI
comeintothisroom,anyway?”(Inaddition,theproblemisonlymade
worse by /usr/games/fortune!)
Atanyrate,moreeffortspentloggingintoamachinemeanslesseffortspent
solvingproblems.Recentversionsof ssh offerasecurealternativetoend-
lessly entering a password: public key exchange.140 | SSH
#66 Quick Logins with ssh Client Keys
HACK
Tousepublickeyswithan ssh server,you’llfirstneedtogenerateapublic/
private key pair:
$ ssh-keygen -t rsa
Youcanalsouse -tdsa forDSAkeys,or -trsa1 ifyou’reusingProtocolv1.
(And shame on you if you are! Upgrade to v2 as soon as you can!)
After you enter the above command, you should see something like this:
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rob/.ssh/id_rsa):
JusthitEnterthere.Itwillthenaskyouforapassphrase;justhitentertwice
(but read the Security note below). Here’s what the results should look like:
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/rob/.ssh/id_rsa.
Your public key has been saved in /home/rob/.ssh/id_rsa.pub.
The key fingerprint is:
a6:5c:c3:eb:18:94:0b:06:a1:a6:29:58:fa:80:0a:bc rob@localhost
Thiscreatedtwofiles, ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.Tousethiskey-
pair on a server, try this:
$ ssh server "mkdir .ssh; chmod 0700 .ssh"
$ scp .ssh/id_rsa.pub server:.ssh/authorized_keys2
Ofcourse,substituteyourservernamefor server.Itshouldaskforyour
passwordbothtimes.Now,simply sshserver anditshouldlogyouin
automagicallywithoutapassword.Andyes,itwilluseyourshinynewpub-
lic key for scp, too.
Ifthatdidn’tworkforyou,checkyourfilepermissionsonboth ~/.ssh/* and
server:~/.ssh/*.Yourprivatekey(id_rsa)shouldbe0600(andonlybepresent
on your local machine), and everything else should be 0655 or better.
Terrific.Soyoucannow sshserver quicklyandwithaminimumoffuss.Isit
possibletomakeitevenquickertoconnecttomachinesyoufrequently
touch? You bet, just check out “‘Turbo-mode’ ssh Logins” (#67).
Security Concerns
Someconsidertheuseofpublickeysapotentialsecurityrisk.Afterall,one
onlyhastostealacopyofyourprivatekeytoobtainaccesstoyourservers.
While this is true, the same is certainly true of passwords.
Askyourself,howmanytimesadaydoyouenterapasswordtogainshell
accesstoamachine(or scp afile)?Howfrequentlyisitthesamepassword
onmany(orall)ofthosemachines?Haveyoueverusedthatpasswordina
waythatmightbequestionable(onawebsite,onapersonalmachinethatTurbo-mode ssh Logins#67
SSH | 141
HACK
isn’tquiteuptodate,orpossiblywithan ssh clientonamachinethatyou
don’tdirectlycontrol).Ifanyofthesepossibilitiessoundfamiliar,thencon-
siderthatan ssh keyinthesamesettingwouldmakeitvirtuallyimpossible
foranattackertolatergainunauthorizedaccess(providing,ofcourse,that
you keep your private key safe).
See Also:
• SSH: The Definitive Guide (O’Reilly)
•“‘Turbo-mode’ ssh Logins” (#67)
•“Running the ssh-Agent in a GUI” [Hack #69]
HACK
#67
Turbo-mode ssh Logins Hack #67
Even faster logins from the command line
Ifyou’vejustcomefromtheprevioushack,you’veonlyseenhalfofthesolu-
tion!Evenwithclientkeys,youstillhavetoneedlesslytype sshserver every
timeyouwantto ssh in.Backinthedark,insecure,unenlighteneddaysof
rsh,therewasanobscurefeaturethatIhappenedtolovethathasn’t(yet)
beenportedto ssh.Itusedtobepossibletosymlink /usr/bin/rsh toafileof
thesamenameasyourserver,and rsh wassmartenoughtorealizethatifit
wasn’t called as rsh, that it should rsh to whatever name it was called as.
Ofcourse,thisistrivialtoimplementinshell.Createafilecalled ssh-to with
these two lines in it:
#!/bin/sh
ssh `basename $0` $*
(Thosearebackticksaround basename$0.)NowputthatinyourPATH(if
~/bin doesn’texistorisn’tinyourPATHalready,itshouldbe)andsetup
symlinks to all of your favorite servers to it:
$ cd bin
$ ln -s ssh-to server1
$ ln -s ssh-to server2
$ ln -s ssh-to server3
Now,to ssh toserver1(assumingyou’vecopiedyourpublickeyoveras
describedpreviously)youcansimplytype“server1”andyou’llmagically
endupwithashellonserver1,withouttyping“ssh,”andwithoutentering
yourpassword.That $* attheendallowsyoutorunarbitrarycommandsin
a single line (instead of spawning a shell), like this:
server1 uptime
Download from Wow! eBook <www.wowebook.com>142 | SSH
#68 Using ssh-Agent Effectively
HACK
(Thiswillsimplyshowtheuptime,numberofusers,andloadaverageon
server1,thenexit.Wrapitina for loopanditerateoveralistofserversto
get a pinpoint status of all of your machines.)
Ibelievethatthisisthequickestwayofusing ssh,shortofsettingupsingle
characteraliasestodoitforyou(whichisexcessivelyhackish,unmaintain-
able, and unnecessary, although it does seem to impress some people):
$ alias a='ssh alice'
$ alias b='ssh bob'
$ alias e='ssh eve'
...
Atanyrate, ssh hasmanyoptionsthatmakeitaveryflexibletoolforsecurely
navigatingbetweenanynumberofservers.Nowifwecanjustfindawayto
make it log in and actually fix the machine for you, we’d have a real hack.
See also:
•“Quick Logins with ssh Client Keys” [Hack #66]
•“Using ssh-Agent Effectively” [Hack #68]
HACK
#68
Using ssh-Agent Effectively Hack #68
Use ssh-agent to automatically manage your ssh client keys
The ssh-agentisaveryhandycomponentof ssh thatmanagesyourprivate
keys for you, passing your credentials along whenever they are required.
The ssh-agent manpage tells us the following:
ssh-agentisaprogramtoholdprivatekeysusedforpublickeyauthentica-
tion(RSA,DSA).Theideaisthat ssh-agentisstartedinthebeginningofan
X-sessionoraloginsession,andallotherwindowsorprogramsarestarted
asclientstothe ssh-agentprogram.Throughuseofenvironmentvariables
theagentcanbelocatedandautomaticallyusedforauthenticationwhenlog-
ging in to other machines using ssh(1).
Practically,thismeansthatwithanagentrunning(andwithproperlyconfig-
ured ssh clients),itispossibleto ssh tomultiplemachineswithoutrequiring
acopyofyourprivatekeyoneachinterveningmachine(ortypinginyour
password on every connection).
Assumewealreadyhaveanauthorized ssh key(see“QuickLoginswithssh
ClientKeys” [Hack#66])installedoneachhomer,bart,andlisa.Ifyou ssh to
each machine from your local machine, there’s no problem:
rob@caligula:~$ ssh homer
rob@homer:~$ exit
logoutUsing ssh-Agent Effectively#68
SSH | 143
HACK
Connection to homer.oreillynet.com closed.
rob@caligula:~$ ssh bart
rob@bart:~$ exit
logout
Connection to bart.oreillynet.com closed.
rob@caligula:~$ ssh lisa
rob@lisa:~$ exit
But what happens when we try to ssh from homer directly to bart?
rob@caligula:~$ ssh homer
rob@homer:~$ ssh bart
rob@bart's password:
Thisiswhere ssh-agentcomesinhandy.Ratherthanexposeyourprivatekey
tounnecessaryriskbyplacingacopyonallofyourservers,simplystartthe
agent on your local machine like this:
rob@caligula:~$ eval `ssh-agent`
Agent pid 8450
Then add your default ssh keys with the ssh-add command:
rob@caligula:~$ ssh-add
Identity added: /home/rob/.ssh/id_rsa (/home/rob/.ssh/id_rsa)
Identity added: /home/rob/.ssh/id_dsa (/home/rob/.ssh/id_dsa)
Identity added: /home/rob/.ssh/identity (rob@caligula)
You’llalsoneedtocheckthathomer,bart,andlisaareconfiguredtofor-
wardagentrequestsalong.Thisisusuallydeniedbydefault,butiscon-
trolled with a line like this:
ForwardAgent yes
inyour ~/.ssh/config or /usr/local/etc/ssh_config file.Youcanalsospecifyit
from the command line with the -A switch.
Now,whenyou ssh fromhomerdirectlytobart,homerwillfirstaskyour
agentforanyavailablecredentials.Likewise, sshingfrombarttolisawillfirst
causebarttocheckwithhomer,whowillforwardtherequestbacktoyour
agent. This makes it easy to skate from machine to machine very quickly:
rob@caligula:~$ ssh homer
rob@homer:~$ ssh bart
rob@bart:~$ ssh lisa
rob@lisa:~$
Congratulations.Younowhaveverysimplenetworknavigation,without
theworryofgivingawayyourprivate ssh keys.Evencopyingfilesbetween
machines (with scp) is faster and easier than before.
Butwhathappensifyouhavenoopportunitytostartanagentinthefirst
place?(ThisisfrequentlythecaseifyourloginisagraphicalXDMprompt,
orifyou’rerunningOSX.)Fearnot.Atleast,notbeforereading“Running
the ssh-Agent in a GUI” [Hack #69].144 | SSH
#69 Running the ssh-Agent in a GUI
HACK
Surely,thereisnowaytomakeitevenfasterandeasierto ssh toamachine,
withoutmakingitlesssecure.Oristhere?Fortheanswer,checkout
“‘Turbo-mode’ ssh Logins” (#67).
HACK
#69
Running the ssh-Agent in a GUI Hack #69
Running ssh-agent in windowing environments
This ssh-agentmethoddescribedin“Usingssh-AgentEffectively” [Hack#68]
worksfineaslongasthatinitialevalssh-agentcanberunbeforeyourwin-
dowenvironmentstarts.Oneplacetodothisisatlogin,inyour ~/.bash_
login (or ~/.login ifrunning tcsh).Butthiscanbealessthanoptimalsolu-
tionifyou’reusingpassphrasesonyourkeys(whichyouprobablyshould
be).Yourwindowsystemwon’tstartuntilyouenteryourkeypasswords,
afterloggingin.Anothersideeffecttothisapproachisthatifyour ssh-agent
evergetskilled,you’llhavetoquitX,startanewagent,andloginagain.
Andinsomeenvironments(suchasOSX)thereisneverevenanopportu-
nity to run commands before the graphical environment starts.
Ratherthanwastefullyrunninganagentforeverywindowyouhaveopen
(orcopy-and-pastingtheenvironmentsettingsbetweenthem),trythiscode
in your ~/.profile:
if [ -f ~/.agent.env ]; then
. ~/.agent.env > /dev/null
if ! kill -0 $SSH_AGENT_PID > /dev/null 2>&1; then
echo "Stale agent file found. Spawning new agent..."
eval `ssh-agent | tee ~/.agent.env`
ssh-add
fi
else
echo "Starting ssh-agent..."
eval `ssh-agent | tee ~/.agent.env`
ssh-add
fi
Thiswillmaintaina ~/.agent.env fileforyou,withanenvironmentpointing
atyourcurrentlyrunning ssh-agent.Iftheagentdies,openinganewtermi-
nalwindowwillspawnoneautomatically(andaddyourkeysforyou),
which is shared between all subsequent terminal windows.
Ahh, that’s better. A single, respawning ssh-agent on demand.
See Also:
•“Quick Logins with ssh Client Keys” [Hack #66]
•“‘Turbo-mode’ ssh Logins” (#67)X over ssh#70
SSH | 145
HACK
HACK
#70
X over ssh Hack #70
Run remote X11 applications easily and securely with ssh
Surprisinglyfewpeoplerealizethat ssh isperfectlycapableofforwarding
X11traffic.IfX11forwardingispermittedbythe ssh serverthatyou’relog-
ging into, starting X applications is as simple as:
rob@florian:~$ ssh -X catlin
Last login: Thu Sep 5 22:59:25 2002 from florian.rob.nocat
Linux 2.4.18.
rob@catlin:~$ xeyes &
[1] 12478
rob@catlin:~$
Aslongasyou’rerunningXonyourlocalmachine,thiswilldisplayxeyeson
yourdesktop.Thisxeyesisactuallyrunningoncatlin,themachinewe’recur-
rentlyloggedinto.AllX11trafficisbeingencryptedandsentdownthe ssh con-
nection that we’re logged in under and is displayed locally.
The real work is done by ssh, which sets up a local X11 proxy server for you:
rob@catlin:~$ echo $DISPLAY
catlin:10.0
X11forwardingisnormallydisabledbydefaultinopenssh.Toenableit,add
the following line to your sshd_config, and restart sshd:
X11Forwarding yes
Whilexeyesisn’tthemostusefulexampleofwhyyouwouldwanttodo
this, here are a couple you might find more interesting:
ethereal
Does packet capturing and visual analysis on a server at your co/lo
vnc
TakescommandofremoteXdesktopsfromyourlocalterminal,asif
you were sitting at the console
gkrellm
Showsanicegraphicalsystemstatusforyourserver(orevenseveralata
time)
TomakeX11trafficforwardover ssh automatically,trysettingthisinyour
~/.ssh/config:
ForwardX11 yes
Withthisoptionineffect,youwon’tneedthe -X switchanymore.Combine
thiswiththe ssh-to script(asdescribedin“‘Turbo-mode’sshLogins” (#67)),
and you have a very quick way of firing off secure, remote X11 jobs:
rob@florian:~$ catlin ethereal &146 | SSH
#71 Forwarding Ports over ssh
HACK
Thiswillinstantlygiveyouan ethereal sessionrunningoncatlin,encrypted
over ssh.Hanginguponthe ssh connection(say,with~.)willkillallrun-
ningXapplicationsinstantly.Butifyouneedtopausethemtemporarily,
suspendingyour ssh session(with~^Z)andresuminglaterwith fg works
just fine.
See also:
•Ethereal packet sniffer: http://www.ethereal.com/
•VNC (a free X remote desktop client): http://www.uk.research.att.com/vnc/
•gkrellm system monitor: http://www.gkrellm.net/
HACK
#71
Forwarding Ports over ssh Hack #71
Keep network traffic to arbitrary ports secure with ssh port forwarding
Inadditiontoprovidingremoteshellaccessandcommandexecution,
OpenSSHcanforwardarbitraryTCPportstotheotherendofyourconnec-
tion.Thiscanbeveryhandyforprotectingemail,web,oranyothertrafficyou
need to keep private (at least, all the way to the other end of the tunnel).
ssh accomplisheslocalforwardingbybindingtoalocalport,performing
encryption,sendingtheencrypteddatatotheremoteendofthe ssh connec-
tion,thendecryptingitandsendingittotheremotehostandportyouspec-
ify. Start an ssh tunnel with the -L switch (short for Local):
root@laptop:~# ssh -f -N -L110:mailhost:110 -l usermailhost
Naturally,substitute user withyourusername,and mailhost withyour
mailserver’snameorIPaddress.Notethatyouwillhavetoberootonlap-
topforthisexample,sinceyou’llbebindingtoaprivilegedport(110,the
POPport).YoushouldalsodisableanylocallyrunningPOPdaemon(look
in /etc/inetd.conf) or it will get in the way.
NowtoencryptallofyourPOPtraffic,configureyourmailclienttocon-
necttolocalhostport110.Itwillhappilytalktomailhostasifitwerecon-
nected directly, except that the entire conversation will be encrypted.
The -f forks ssh intothebackground,and -N tellsitnottoactuallyruna
commandontheremoteend(justdotheforwarding).Ifyour ssh serversup-
portsit,trythe -C switchtoturnoncompression—thiscansignificantly
improve the time it takes to download your email.
Youcanspecifyasmany -L linesasyoulikewhenestablishingtheconnec-
tion. To also forward outbound email traffic, try this:
root@laptop:~# ssh -f -N -L110:mailhost:110 -L25:mailhost:25 \
-l usermailhostForwarding Ports over ssh#71
SSH | 147
HACK
Setyouroutboundemailhosttolocalhost,andyouremailtrafficwillbe
encryptedasfarasmailhost.Thisgenerallyisonlyusefuliftheemailis
boundforaninternalhost,orifyoucan’ttrustyourlocalnetworkconnec-
tion(asisthecasewithmostwirelessnetworks).Obviously,onceyour
emailleavesmailhost,itwillbetransmittedintheclear,unlessyou’ve
encrypted the message with a tool such as pgp or gpg.
Ifyou’realreadyloggedintoaremotehostandneedtoforwardaport
quickly, try this:
•Hit Enter
•Type ~C
•Youshouldbeatanssh>prompt;enterthe -L lineasyouwouldfrom
the command line.
For example:
rob@catlin:~$
rob@catlin:~$ ~, then C (it doesn't echo)
ssh> -L8080:localhost:80
Forwarding port.
Yourcurrentshellwillthenforwardlocalport8000tocatlin’sport80,asif
you had entered it in the first place.
Youcanalsoallowother(remote)clientstoconnecttoyourforwardedport,
withthe -g switch.Ifyou’reloggedintoaremotegatewaythatservesasa
NAT for a private network, then a command like this:
rob@gateway:~$ ssh -f -g -N -L8000:localhost:80 10.42.4.6
willforwardallconnectionsfromgateway’sport8000tointernalhost
10.42.4.6’sport80.IfthegatewayhasaliveInternetaddress,thiswill
allowanyonefromtheNettoconnecttothewebserveron10.42.4.6asif
it were running on port 8000 of the gateway.
Onelastpointworthmentioning:theforwardedhostdoesn’thavetobe
localhost;itcanbeanyhostthatthemachineyou’reconnectingtocan
accessdirectly.Forexample,toforwardlocalport5150toawebserver
somewhere on an internal network, try this:
rob@remote:~$ ssh -f -N -L5150:intranet.insider.nocat:80 gateway.nocat.net
Assumingthatyou’rerunningaTLD(“RunningYourOwnTop-Level
Domain” [Hack#80])of .nocat,andthat gateway.nocat.net alsohasaconnection
totheprivate .nocat network,alltrafficto5150of remote willbeobligingly
forwardedto intranet.insider.nocat:80.Theaddress intranet.insider.nocat148 | SSH
#71 Forwarding Ports over ssh
HACK
doesn’thavetoresolveinDNSto remote;itisn’tlookedupuntiltheconnec-
tionismadeto gateway.nocat.net,thenit’sgatewaythatdoesthelookup.To
securely browse that site from remote, try connecting to http://localhost:5150/.
Although ssh alsohasfunctionalityforactingasaSocks4proxy(withthe
-D switch),itjustisn’twellsuitedforroutingallnetworktraffictothe
otherendofatunnel.Takealookat“Tunneling:IPIPEncapsulation” [Hack
#50] forawaytouse vtun inconjunctionwith ssh toforward everything.
Andseethedocumentationforthe -D switch;it’saprettyneatfeature.
(What, did you think we’d do all of the work for you? ;)
ssh isanincrediblyflexibletool,withmuchmorefunctionalitythanIcan
coverhere.Seethereferencesbelowformorefunthingsyoucandowith ssh.
See also:
•man ssh
• SSH, The Secure Shell: The Definitive Guide (O’Reilly)
•“Tunneling: IPIP Encapsulation” [Hack #50]
•“Quick Logins with ssh Client Keys” [Hack #66]
•“Running Your Own Top-Level Domain” [Hack #80]149
Chapter7 CHAPTER SEVEN
Scripting
Hacks #72–75
Sometimes,youwanttodosomethingmorecomplicatedthancanbe
expressedonasinglecommandline.Onceyou’vedonesomethingcompli-
catedmorethantwoorthreetimes,you’llprobablybegintothinkthatthere
mustbeabetterwaytodoit(whichdoesn’tinvolveawholebunchoftyp-
ingeachtime).Thisisusuallywhensysadminsbegintoevolveintothat
strange breed of problem solver known as the programmer.
Thissectionis not acrashcourseinprogramming,butratherademonstra-
tionofacoupleofpiecesofappliedprogramminghackery.I’vefoundthat
thebestwaytolearnhowtoprogramistolearnbydoing,andtheeasiest
waytogetstartedistoseehowotherpeoplehavesolvedsimilarproblems.
Theexamplesinthissectionareusefulontheirownbutareevenmoreuse-
fulasastartingpointinbuildingyourowncustomtools.Evenifyou’rean
oldhandatscripting,takealookatsomeoftheseexamplesforsomeideas
onhowtousesomelesserknowninvocationswitchesandlanguagefea-
tures to get more done with less effort.
HACK
#72
Get Settled in Quickly with movein.sh Hack #72
Keep your local environment in sync on all of your servers
Whenyouuseamachineforsometime,youwillinevitablyendupcustom-
izingittoyourliking.Aswesawin“AtHomeinYourShellEnvironment”
[Hack#10],theshellenvironmentisatremendouslyflexibletoolthatcanbe
fine tuned to your precise specifications.
Usuallytheselittletweakstakeweeks(orevenyears)toperfectandspan
severalfiles:differentenvironmentsettings(dependingonifit’saloginshell
orasubshell),editorpreferences,emailsettings,mysqlpreferences,aliases
tobesetandoptionsforeveryoccasion,etc.Whentheenvironmentispre-
ciselyprogrammed,tasksbecomeeasierandthesystembecomesmorefun
to work with.
Download from Wow! eBook <www.wowebook.com>150 | Scripting
#72 Get Settled in Quickly with movein.sh
HACK
Ofcourse,allofthisgoesoutthewindowwhenyoulogintoaremote
machine.Itcanbeterriblyfrustratingtofireoffyourfavoritecommand,only
toseebash:xyz:commandnotfound.Ortorun ls andnotseethecolors.Or
anyofathousandlittleadjustmentsthatmakelifeonyourhomemachine
such a joy, but working on remote machines such a barren, joyless existence.
Youcanalwayscopyoveryoursettingsbyhand,asmostofthemarekeptas
dotfilesinyourhomedirectory(like .bashrc and .vimrc).Butit’sdifficultto
rememberthemallbyhand,anditleadstotheclassicversioningproblem:
onceyoumakeachangetoyourlocalcopy,theremotecopiesallneedtobe
updated again.
Make your life simpler with this very simple shell script.
Listing: movein.sh
#!/bin/sh
if [ -z "$1" ]; then
echo "Usage: `basename $0` hostname"
exit
fi
cd ~/.skel
tar zhcf - . | ssh $1 "tar zpvxf -"
Callit movein.sh,andstickitinyour ~/bin directory.Nowcreatea ~/.skel
directory,andmakesymlinkstoallofthefilesyou’dliketocopytoremote
servers:
rob@caligula:~/.skel$ ls -al
total 12
drwxr-xr-x 6 rob staff 204 Sep 9 20:52 .
drwxr-xr-x 37 rob staff 1258 Sep 9 20:57 ..
lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .bash_login -> ../.bash_login
lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .bashrc -> ../.bashrc
lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .my.cnf -> ../.my.cnf
lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .pinerc -> ../.pinerc
drwxr-xr-x 3 rob staff 102 Sep 9 20:51 .ssh
lrwxr-xr-x 1 rob staff 9 Sep 9 20:52 .vimrc -> ../.vimrc
lrwxr-xr-x 1 rob staff 6 Sep 9 21:27 bin -> ../bin
Notethat ~/.skel/.ssh isaspecialcase:it’sadirectory,notasymlink.DO
NOTSYMLINKYOUR ~/.ssh TO ~/.skel!Thelastthingyouneedisto
copyyourprivate ssh keyallovertheplace;that’swhatthe ssh-agentis
for(see“Usingssh-AgentEffectively” [Hack#68]).Instead,makeadirectory
called ~/.skel/.ssh and make a symlink like this:
rob@caligula:~/.skel$ cd .ssh
rob@caligula:~/.skel/.ssh$ ls -al
total 4Global Search and Replace with Perl#73
Scripting | 151
HACK
drwxr-xr-x 3 rob staff 102 Sep 9 20:51 .
drwxr-xr-x 6 rob staff 204 Sep 9 20:52 ..
lrwxr-xr-x 1 rob staff 26 Sep 9 20:51 authorized_keys2 ->
../../.ssh/id_dsa.pub
Thisisalinkcalled authorized_keys2,anditpointstoyourlivepublickey.
You are usingpublickey ssh connections,right?Ifnot,consult“QuickLog-
ins with ssh Client Keys” [Hack #66].
Nowwhenthisscriptruns,itwillcopythecontentsof ~/.skel tothehost
youspecifyonthecommandline,straightintoyourhomedirectory.The h
flagtotarmeans“copythesesymlinksasiftheywerefiles,notsymlinks,”so
youendupwithacopyofthecontentsofeachlinkontheremoteend.If
youmakechangestoyourlocalcopy,justrunthescriptagain,anditwill
overwrite everything on the remote end.
Onlyincludethe .ssh directoryasmentionedaboveifyou’dliketobeableto
logintotheremotehostautomatically,withoutapassword.Aslongasyour
localmachine’skeysarekeptsecure,thereisnoinherentsecurityriskin
leavingextra authorized_keys2 fileslyingaround.That’swhatpublickey
cyptography is all about.
See also:
•“Quick Logins with ssh Client Keys” [Hack #66]
•“Using ssh-Agent Effectively” [Hack #68]
•“At Home in Your Shell Environment” [Hack #10]
• SSH: The Secure Shell, The Definitive Guide (O’Reilly)
HACK
#73
Global Search and Replace with Perl Hack #73
ManipulatefilesandstreamswitharbitraryPerlsubstitutions,withoutascript
ThereareacoupleofswitchesthatmakePerlaveryusefulcommand-line
editingtool.Learntheseswitches,andyoutoocanlearnhowtomumble
magicPerlone-linerstoconfoundyourfriends(andfrightenyourproject
manager).
Thefirstis -e.GivePerla -e followedbyalineofcode,anditwillrunitasif
it were an ordinary Perl script:
rob@catlin:~$ perl -e 'print "Hi, Ma!\n"'
Hi, Ma!
Notethatatrailing ; isn’tneededonone-liners.It’sgenerallyagoodideato
wrapyourlineinsinglequotestopreventtheshellfromattemptingtointer-
pret special characters (like the ! and \ above).152 | Scripting
#73 Global Search and Replace with Perl
HACK
Thenextswitchisalittlemorecomplicated,itbutbecomessecondnature
once you start using it: -p. From perldoc perlrun:
-p causes Perl to assume the following loop around your
program, which makes it iterate over filename arguments
somewhat like sed:
LINE:
while (<>) {
... # your program goes here
} continue {
print or die "-p destination: $!\n";
}
Thelineinquestion($_)isautomaticallyprintedoneveryiterationofthe
loop.Ifyoucombine -p and -e,yougetaone-linerthatiteratesoverevery
fileonthecommandlineoronthetextfedtoitonSTDIN.Forexample,
here’s a complicated cat command:
rob@catlin:~$ perl -pe 1 /etc/hosts
#
# hosts This file describes a number of hostname-to-address
# mappings for the TCP/IP subsystem.
#
# For loopbacking.
127.0.0.1 localhost
The 1 isjustareturncode(asifyouentered 1; inaPerlscript,whichis
equivalentto return1;).Sincethelinesareprintedautomatically,wedon’t
really need the program we specify with -e to do anything.
Whereitgetsinterestingisinprovidingabitofcodetomanipulatethecur-
rentlinebeforeitgetsprinted.Forexample,supposeyouwantedtoappend
the local machine name to the localhost line:
rob@catlin:~$ perl -pe 's/localhost/localhost $ENV{HOSTNAME}/' /etc/hosts
#
# hosts This file describes a number of hostname-to-address
# mappings for the TCP/IP subsystem.
#
# For loopbacking.
127.0.0.1 localhost catlin.nocat.net
or maybe you’d like to manipulate your inetd settings:
rob@caligula:~$ perl -pe 's/^(\s+)?(telnet|shell|login|exec)/# $2/' \
/etc/inetd.conf
Thatwillprintthecontentsof /etc/inetd.conf toSTDOUT,commentingout
anyuncommented telnet, shell, login,or exec linesalongtheway.Naturally,
wecouldredirectthatbackouttoafile,butifwejustwanttoeditafilein
place, there’s a better way: the -i switch.Mincing Your Data into Arbitrary Chunks (in bash)#74
Scripting | 153
HACK
-i letsyoueditfilesinplace.So,tocommentoutalloftheabovelinesin /
etc/inetd.conf, you might try:
root@catlin:~# perl -pi -e 's/^(\s+)?(telnet|shell|login|exec)/# $2/' /etc/
inetd.conf
or better yet:
root@catlin:~# perl -pi.orig -e 's/^(\s+)?(telnet|shell|login|exec)/# $2/' /
etc/inetd.conf
Thesecondexamplewillbackup /etc/inetd.conf to /etc/inetd.conf.orig before
changingtheoriginal.Don’tforgettoHUP inetd tomakeyourchangestake.
It’sjustaseasytoeditmultiplefilesinplaceatthesametime.Youcanspec-
ify any number of files (or wildcards) on the command line:
rob@catlin:~$ perl -pi.bak -e 's/bgcolor=#ffffff/bgcolor=#000000/i' *.html
Thiswillchangethebackgroundcolorofallhtmlpagesinthecurrentdirec-
toryfromwhitetoblack.Don’tforgetthattrailing i tomakethematchcase
insensitive (to match bgcolor=#FFFFFF or even BGColor=#FfFffF).
Whatifyou’reinthemiddleofworkingonaCVSproject,andneedto
changetheCVSserverthatyou’dliketocommitto?It’seasy,ifyoupipethe
output of a find through an xargs running a perl -pi -e:
schuyler@ganesh:~/nocat$ find -name Root | xargs perl -pi -e 's/cvs.
oldserver.com/cvs.newserver.org/g'
Thenresetyour$CVSROOTanddoyourCVScheckinasnormal,andyour
project will automagically end up checked into cvs.newserver.org.
UsingPerlfromthecommandlinecanhelpyoudosomepowerfultransfor-
mationsonthefly.Studyyourregularexpressions,anduseitwisely,andit
can save piles of hand edits.
See also:
• Programming Perl (O’Reilly)
•perldoc perlrun
•“At Home in Your Shell Environment” [Hack #10]
•“CVS: Making Changes to a Module” [Hack #30]
HACK
#74
Mincing Your Data into Arbitrary Chunks (in
bash) Hack #74
Use bash arithmetic and dd to chop large binary files into reasonable chunks
Here’sanexampleofhowtouseenvironmentvariablesandarithmeticeval-
uation in bash to chop any file into arbitrary sized chunks.154 | Scripting
#74 Mincing Your Data into Arbitrary Chunks (in bash)
HACK
Listing: mince
#!/bin/bash
if [ -z "$2" -o ! -r "$1" ]; then
echo "Usage: mince [file] [chunk size]"
exit 255
fi
SIZE=`ls -l $1 | awk '{print $5}'`
if [ $2 -gt $SIZE ]; then
echo "Your chunk size must be smaller than the file size!"
exit 254
fi
CHUNK=$2
TOTAL=0
PASS=0
while [ $TOTAL -lt $SIZE ]; do
PASS=$((PASS + 1))
echo "Creating $1.$PASS..."
dd conv=noerror if=$1 of=$1.$PASS bs=$CHUNK skip=$((PASS - 1)) count=1 2> /
dev/null
TOTAL=$((TOTAL + CHUNK))
done
echo "Created $PASS chunks out of $1."
Notethatwetakeadvantageofconv=noerror,sincethelastchunkisalmost
guaranteedtorunbeyondtheendofyourfile.Usingthisoptionmakes dd
blithelycontinuetowritebitsuntilwerunoutofsourcefile,atwhichpoint
itexits(butdoesn’tthrowanerror,andmoreimportantly,doesn’trefuseto
write the last chunk).
Thiscouldbehandyforslicinglargefilesintofloppy-sized(zipdisk,cd-r,
dvd,Usenet)chunkspriortoarchiving.Asituses dd’sskipfeature,itwill
workonanysizedfile,regardlessoftheamountofavailableRAM(provided
thatyousupplyareasonablechunksize).Sincetheblocksize(bs)issetto
whateveryouhaveselectedasyourchunksize,itrunsquitequickly,espe-
cially with chunks larger than a couple of kilobytes.
Itsavesyourchunksasmultiplefiles(endingina.followedbythechunk
number)inthecurrentdirectory.Running lsFILENAME.* willshowyouyour
chunks in numerical order. But what if you have more than nine of them?
ls FILENAME.* | sort -n -t . +2
How do you reassemble them?
cat `ls FILENAME.* | sort -n -t . +2` > FILENAME.completeColorized Log Analysis in Your Terminal#75
Scripting | 155
HACK
Don’t believe me?
diff FILENAME FILENAME.complete
(Thisofcourseassumesthatyourfilenamehasonlyone.init.Ifyouhavea
ridiculously.long.filename.with.multiple.dots, consult man sort(1).)
HACK
#75
Colorized Log Analysis in Your Terminal Hack #75
View your log files in an xterm window in full, living color
Ifyoufindyourselfslowlygoingcross-eyedwhilelookingatlineafterlineof
systemlogs,thenyoushouldconsiderusingtoolstohelpyouorganizeyour
logs.Whileaproperlyconfiguredsyslog(asshownin“Steeringsyslog” [Hack
#54])goesalongwaytowardlogfilesanity,itcanstillbeabitoverwhelming
to sift through a multi-megabyte /var/log/messages looking for patterns.
Justasacolorized ls canhelpidentifytypesoffilesataglance,acolorized
grep isahandytoolformakingpatternsleapoutofaseaofgreylines.There
areanumberofXapplicationsthatwilldothisforyou,butwhynotmakeit
easy to view your colorized logs from the command line?
Save this as ~/bin/rcg (short for Regex Colored Glasses):
#!/usr/bin/perl -w
use strict;
use Term::ANSIColor qw(:constants);
my %target = ( );
while (my $arg = shift) {
my $clr = shift;
if(($arg =~ /^-/) | (!$clr)) {
print "Usage: rcg [regex] [color] [regex] [color] ...\n";
exit;
}
#
# Ugly, lazy, pathetic hack here
#
$target{$arg} = eval($clr);
}
my $rst = RESET;
while(<>) {
foreach my $x (keys(%target)) {
s/($x)/$target{$x}$1$rst/g;
}
print;
}156 | Scripting
#75 Colorized Log Analysis in Your Terminal
HACK
rcg isasimplefilterthatusesTerm::ANSIColortocolorizearbitraryregexs,
specifiedonthecommandline.Itisintendedtohelpvisuallyslogthrough
log files.
Youmustpass rcg anevennumberofcommandlineparameters.Theodd
terms specify the regex, the even terms specify the color.
Supposeyouwantedanythingwiththeword sendmail inyourmessageslog
to show up magenta, instead of grey:
$ rcg sendmail MAGENTA < /var/log/messages | less -r
The less-r isoptionalbuthandy(asitdisplaystheintendedcolorsin less,
instead of the ESC characters.)
You can use any arbitrary regex as an odd term:
$ rcg '\d+\.\d+\.\d+\.\d+' GREEN < /var/log/maillog
Or chain colors together:
$ tail -50 /var/log/messages | rcg WARNING 'BOLD . YELLOW . ON_RED'
Youcanspecifyanynumberofregex/colorcodepairsonasinglecommand-
line.Thisiswhereteenyshellscriptsoraliaseswouldcomeinhandy(one
for messages, one for firewall logs, one for Apache).
See the Term::ANSIColor docs for the full list of colors and combinations.
Some other useful strings:
\w+=\S+
Variables, such as TERM=xyz
\d+\.\d+\.\d+\.\d+
Probably an IP address
^(J|F|M|A|S|O|N|D)\w\w (\d|)\d
Might be a date
\b\d\d:\d\d:\d\d\b
Possibly the time
.*last message repeated.*
Makes this “BOLD . WHITE”
Useyourimagination,butbewarned:colorparamsarejusteval()'d.Theo-
retically,manyvalidPerlexpressionscanbesubstitutedforregexesorcol-
ors;exploitingthis,uh, feature isleftasanexercisetothereader.You
probablyshouldn’tberunningarbitrary rcg linesasroot,unlessyouwrote
themyourself.Alsonotethatcolorizationisappliedinarbitraryorder,soit’s
not possible to guarantee the behavior of overlapping regexes.157
Chapter8 CHAPTER EIGHT
Information Servers
Hacks #76–100
Linuxisapowerfulplatformforbuildinginformationservers.Buttheinfor-
mationsystemsthemselvesarerarelyapartofLinux.Usually,Linuxissim-
plya“lifesupportsystem”formorecomplicated,dedicatedinformation
services.
Inthisfinalsection,we’lllookatthreemajorapplications.Theyallrunquite
wellonLinuxandhavebecomethebackboneofInternetinformationser-
vices.BIND(bytheInternetSoftwareConsortium)isbyfarthemostcom-
monDNSinformationserverontheplanet,servingtheDomainNametoIP
addressinformationthatkeepstheInternetrunning.Formoregenericinfor-
mationneeds,MySQLisaverylightweight,fast,andscalableSQLdatabase
thatdrivesmanyenterprisenetworkapplications.Finally,whenitcomesto
servinginformationtousers,Apacheisbyfarthemostpopularwebserver
ontheplanet.Apacheisrunonmoreserversthanalloftherestofthe
knownwebservers combined andforgoodreason:itismature,stable,fast,
and full of useful and interesting features.
Ifyou’relookingformoreinformationaboutrunninganyofthesepack-
ages,consulttheonlinedocumentationforeach.Theseapplicationsarerun-
ningthecurrentincarnationoftheInternet,andhavebeenwidelyand
extensivelydocumented.Inaddition,O’Reillyhasanumberofgoodbooks
onallthreeapplications,suchas DNSandBind, MySQLReferenceManual,
and Apache:TheDefinitiveGuide tonameafew. MySQL byPaulDuBois
(New Riders) is also an excellent guide to running MySQL.
Inthefollowinghacks,we’llseesomenon-obvioustechniquesforgetting
theseserverstodeliverinformationthewayyouwantthem.Wewilltakea
lookatsomemethodsformakingthemscaletolargeinstallations,while
keeping maintenance of even very complex sites to a minimum.158 | Information Servers
#76 Running BIND in a chroot Jail
HACK
HACK
#76
Running BIND in a chroot Jail Hack #76
Keepyournamedisolatedfromtherestofthesystemwiththejudicioususe
of chroot
ThevastmajorityoftheInternetreliesonBINDforitsnameresolutionneeds.
Whiletremendousefforthasgonetowardshoringuppotentialsecurityholes
inBIND,youcanneverbeabsolutelycertainthatanycodeiscompletelyfree
ofpossibleexploits.Tominimizethepossibledamagedonebyremoteroot
exploits(duetobufferoverflows,bugs,ormisconfiguration),manysites
choosetoruntheir named serviceinachrootjail.Thishelpsensurethateven
ifthe named processiscompromised,theattacker’sjobwon’tbefinishedyet.
Whileachrootjailisn’tnecessarilyimpenetrable,itposesaverydifficultchal-
lenge for a would-be system cracker.
ThesestepsoutlinetheminimumeffortrequiredtogetBIND9runningin
achrootjail.DNSsecurityisalargeandcomplexissue,andshouldbe
takenveryseriously.Consulttheresourcesattheendofthishackformore
information.
Tobeginwith,we’llwanttorun named assomeuserotherthanroot.Cre-
ateanameduserandgroupthatwillonlybeusedforrunningthe named
process:
root@gemini:~# groupadd -g 53 named
root@gemini:~# useradd -u 53 -g named -c "chroot BIND user" \
-d /var/named/jail -m named
We’llinstruct named to chroot to /var/named/jail atstartup.Wewillneedto
createenoughofaskeletonfilestructureunderthisdirectorytoallow named
to start normally. Create the /var structure, and copy your DNS data into it:
root@gemini:~# cd ~named
root@gemini:/var/named/jail# mkdir -p var/{run,named}
root@gemini:/var/named/jail# cp -Rav /var/named/data var/named/
Ifyouactasaslaveforanyzones,thenthe named processwillneedwrite
accesstoadirectory(tokeeptrackofupdatedslavedata).Createadirec-
tory specifically for slave data, independent of your regular data files:
root@gemini:/var/named/jail# mkdir var/named/slave
root@gemini:/var/named/jail# chown named.named var/named/slave
Next,createthe dev/ and etc/ directories,andcopythecriticalsystemfilesand
devices to them:
root@gemini:/var/named/jail# mkdir {dev,etc}
root@gemini:/var/named/jail# cp -av /dev/{null,random} dev/
root@gemini:/var/named/jail# cp -av \
/etc/{localtime,named.conf,rndc.key} etc/Running BIND in a chroot Jail#76
Information Servers | 159
HACK
Clean up the ownership and permissions of these directories:
root@gemini:/var/named/jail# chown root.root .
root@gemini:/var/named/jail# chmod 0755 .
root@gemini:/var/named/jail# chown named.named var/named/data/
root@gemini:/var/named/jail# chmod 0700 var/named/data/
root@gemini:/var/named/jail# chown named.named var/run/
Ifyou’reusingsyslogforyourDNSlogs,you’llneedtoaddaswitchlikethisto
yoursyslogdstartuprc.Itwilltellsyslogtolistenonthatsocketaswell(which
is needed since the system’s /dev/log won’t be reachable from inside the jail):
syslogd -m 0 -a /var/named/jail/dev/log
Ifyou’reusingfilesystemlogs,youwon’tneedtochangeyoursyslogdcon-
figuration.Justbesurethatthelogfilesarewritablebythenameduser,and
that your log directory exists under /var/named/jail/.
Finally,fireup named,passingtheusername,chrootdirectory,andinitial
configuration file on the command line:
root@gemini:~# /usr/sbin/named -u named -t /var/named/jail \
-c /etc/named.conf
Ifyour named doesn’tstartcleanly,takealookatthesystem’s /var/log/syslog
or /var/log/messages,andtrackdownthesourceofthetrouble.Itcanbe
trickytobesurethatyourpermissionsareallwhattheyneedtobe.Here’sa
sample recursive ls output from a machine with a running chroot’d BIND:
root@gemini:/var/named/jail# ls -lR
.:
total 12
drwxr-xr-x 2 root root 4096 Sep 20 12:45 dev/
drwxr-xr-x 2 root root 4096 Sep 20 13:08 etc/
drwxr-xr-x 5 root root 4096 Sep 20 13:04 var/
./dev:
total 0
crw-rw-rw- 1 root root 1, 3 Jul 17 1994 null
crw-r--r-- 1 root root 1, 8 Dec 11 1995 random
./etc:
total 16
-rw-r--r-- 1 root root 1017 Sep 20 12:46 localtime
-r--r--r-- 1 root root 1381 Sep 20 13:08 named.conf
-rw------- 1 root root 77 Sep 11 04:22 rndc.key
./var:
total 12
drwxr-xr-x 4 root root 4096 Sep 20 13:01 named/
drwxr-xr-x 2 named named 4096 Sep 20 13:15 run/
./var/named:
total 8160 | Information Servers
#77 Views in BIND 9
HACK
drwx------ 3 named named 4096 Sep 20 13:03 data/
drwxr-xr-x 2 named named 4096 Sep 20 12:43 slave/
./var/named/data:
total 42
-rw-r--r-- 1 root root 381 Apr 30 16:32 localhost.rev
-rw-r--r-- 1 root root 2769 Sep 20 13:03 named.ca
-r--r--r-- 1 root root 1412 Sep 17 16:44 nocat.net
./var/named/slave:
total 0
./var/run:
total 4
-rw-r--r-- 1 named named 6 Sep 20 13:15 named.pid
A chroot’denvironmentcanhelpkeepdaemonssequesteredawayfromthe
restofarunningsystem,butthey’renotamagicbulletforinstantsystem
security.Thishackshouldgetyoustarted,butbesuretoreaduponthe
complexities of using and running BIND.
See also:
• http://www.losurs.org/docs/howto/Chroot-BIND.html
• DNS Bind, Fourth edition (O’Reilly)
HACK
#77
Views in BIND 9 Hack #77
ChangetheresultsthatyourDNSserverreturnsdependingonwherethe
requests are coming from
Untilrecently,presentingoneviewofazonetoonecommunityofhostsand
anotherviewtoothershasentailedrunningmultiplesetsofnameserversor
multiplenameserverprocessesonasinglehostinatrickyconfiguration.
Nobody ever said being two-faced is easy.
BIND9introducedanewfeaturecalled views thatmakesdeliveringdiffer-
ent versions of a zone and even different name server configurations easy.
Basic Syntax
ThekeytoconfiguringviewsisthenewBIND9configurationstatement,
view.viewtakesaviewnameasanargumentandhasonerequiredsubstate-
ment,match-clients.Theviewnameissimplyaconvenientmnemonicfor
identifyingtheview;Iusuallyusenamessuchas“internal”and“external”
or“Internet.”Ifyouuseanamethatconflictswithareservedwordin
named.conf,besuretoquoteit.Iquoteallofmyviewnames,becauseIcan’t
remember which words are reserved.
Download from Wow! eBook <www.wowebook.com>Views in BIND 9#77
Information Servers | 161
HACK
Thematch-clientssubstatementtakesanaddressmatchlistasanargument.
OnlyquerierswhoseIPaddressesmatchthisaddressmatchlistwillseethe
configurationspecifiedintheenclosingview.Ifaquerier’sIPaddress
matchesmultipleviewstatement’smatch-clientssubstatements,thefirst
view statement is the one that applies.
Here’s an example of a simple view statement:
view "global" {
match-clients { any; };
};
Thisviewstatementdoesn’tdoanythinguseful,becauseitappliestoallque-
ries(intheabsenceofotherviewstatements)anditdoesn’tincludeanysub-
statementsbesidesmatch-clients,whichwouldchangetheconfigurationof
this view from the default configuration.
Ifyoudon’tspecifyaparticularsubstatementwithinaview,theviewinher-
itstheglobalsettingforthatsubstatement.So,forexample,ifyoudon’tturn
recursionoffwithinaview,thatviewwouldinherittherecursionsetting
fromyouroptionsstatement(specifiedbytherecursionsubstatement),orif
you don’t have one, would inherit the global default, which is recursionyes.
Here’s an example of turning recursion off within a view:
view "external" {
match-clients { any; };
recursion no;
};
Here’sacomplete named.conf filethatincludesthepreviousviewstatement:
/*
* BIND 9 name server allowing recursive queries from
* localhost, disallowing from anywhere else
*/
options {
directory "/var/named";
};
view "localhost" {
match-clients { localhost; };
recursion yes; /* this is the default */
};
view "external" {
match-clients { any; };
recursion no;
};
Thisnameserverallowsrecursivequeriesonlyiftheycomefromthelocal
host.(Thelocalhostaccesscontrollist[ACL]ispredefinedtobealloftheIP162 | Information Servers
#77 Views in BIND 9
HACK
addressesofthehostthatrunsthenameserver,plustheloopbackaddress.)
Queries from all other addresses are treated as non-recursive.
Here’sasimilarconfiguration,thisoneofanameserverthatallowsrecur-
sive queries from our internal network but not from the Internet:
/*
* Same name server serving an internal network and the
* Internet
*/
options {
directory "/var/named";
};
view "internal" {
match-clients { localnets; };
recursion yes; /* this is the default */
};
view "external" {
match-clients { any; };
recursion no;
};
Thisconfigurationtakesadvantageofthebuilt-inACLlocalnets,whichispre-
defined as all of the networks to which our name server is directly connected.
Deﬁning Zones in Views
Inordertopresentaversionofazonetoonlysomequeriers,youneedto
definethezonewithinaview.Yousimplyuseazonestatementasaview
substatement.Otherwise,itssyntaxisthesameasifitwereatop-levelstate-
ment.Notethatifyoudefineevenonezonewithinaview,allofyourzones
need to be defined inside views.
Here’s an example:
/*
* A name server showing different zone data to different
* networks
*/
options {
directory "/var/named";
};
view "internal" {
match-clients { localnets; };
recursion yes; /* this is the default */
zone "oreilly.com" {
type master;Views in BIND 9#77
Information Servers | 163
HACK
file "db.oreilly.com.internal";
allow-transfer { any; };
};
};
view "external" {
match-clients { any; };
recursion no;
zone "oreilly.com" {
type master;
file "db.oreilly.com.external";
allow-transfer { none; };
};
};
Notethatthezoneoreilly.comisdefinedinboththeinternalandtheexter-
nalview,butthezonedatafilefororeilly.comisdb.oreilly.com.internalin
theinternalviewanddb.oreilly.com.externalintheexternalview.Presum-
ably, the contents of the zone data files are different.
Ifyouprefertousedifferentsubdirectoriesfortheinternalandexternalzone
data,youcandothat,too.Forexample,thefilesubstatementfororeilly.
comintheinternalviewcouldbe file"internal/db.oreilly.com"; andthe
external view could use file "external/db.oreilly.com";.
Thisway,youcankeepallofyourinternalzonedatafilesinthe /var/named/
internal directoryandyourexternalzonedatafilesinthe /var/named/
external directory.
Views in Slave Name Servers
Oneminorwrinkleinviewsistheconfigurationofslavenameservers.Many
peopleconfigureaprimarymasternameserverwithmultipleviews,then
wanttoconfigureaslavewiththesameviews.Unfortunately,whenthe
slavetriestotransferthezones(orthetwoversionsofthesamezone)from
theprimarymasternameserver,itonlyseesoneversionofthezone,theone
defined in the view the slave’s IP address can see.
ThesolutiontothisproblemistoconfigureanIPaddressaliasontheslave
nameserver,givingittwoIPaddressesfromwhichtoinitiatezonetransfers.
Thenconfiguretheprimarymastertopresentoneviewtooneoftheslave’s
IPaddressesandtheotherviewtotheslave’sotherIPaddress.Finally,force
theslavetoinitiatezonetransfersfromtheappropriateIPaddresswithin
each view.
Here’sanexample.OurslavehastwoIPaddresses,192.168.0.2and192.
168.0.254.Ourprimarymasterhasjustone,192.168.0.1.First,here’sthe
slave’s named.conf file:164 | Information Servers
#77 Views in BIND 9
HACK
options {
directory "/var/named";
};
view "internal" {
match-clients { localnets; };
recursion yes;
zone "oreilly.com" {
type slave;
masters { 192.168.0.1; };
transfer-source 192.168.0.2;
file "internal/bak.oreilly.com ";
allow-transfer { any; };
};
};
view "external" {
match-clients { any; };
recursion no;
zone "oreilly.com" {
type slave;
masters { 192.168.0.1; };
transfer-source 192.168.0.254;
file "external/bak.oreilly.com ";
allow-transfer { none; };
};
};
Noticethattheslaveisconfiguredtoinitiatetransfersoforeilly.comwithin
theinternalviewfromtheIPaddress192.168.0.2andwithintheexternal
view from 192.168.0.254.
Now, here’s the primary master’s named.conf file:
options {
directory "/var/named";
};
view "internal" {
match-clients { !192.168.0.254; localnets; };
recursion yes;
zone "oreilly.com" {
type master;
file "internal/db.oreilly.com ";
allow-transfer { any; };
};
};
view "external" {
match-clients { any; };
recursion no;Setting Up Caching DNS with Authority for Local Domains#78
Information Servers | 165
HACK
zone "oreilly.com" {
type master;
file "external/db.oreilly.com ";
allow-transfer { 192.168.0.254; };
};
};
Noticethattheinternalview’smatch-clientssubstatementexplicitlyplaces
192.168.0.254intotheexternalviewbynegatingitintheaddressmatchlist.
AnytimetheslaveinitiatesazonetransferfromthatIPaddress,it’llgetthe
versionoforeilly.comdescribedbythezonedatafile /var/named/external/db.
oreilly.com.
HACK
#78
Setting Up Caching DNS with Authority for
Local Domains Hack #78
Get BIND running quickly with a forwarding, caching server
RunningBINDcanbeverytrickybusinessifyouhaveaparticularlycom-
plexnetworktopology.MultipleDMZs,publicversusprivateIPaddresses,
anddelegatedsubdomainscanmakeDNSadministrationafulltimejobfor
alargesite.Ifyou’relookingforawaytoalleviatesomeofthecomplexity,
see“ViewsinBIND9” [Hack#77].Orifyou’refeelingparticularlyadventur-
ous,trywildcarddomainmatchinganddelegation,asdescribedin“Ultra-
hosting:MassWebSiteHostingwithWildcards,Proxy,andRewrite” [Hack
#100].
Butinthemajorityofsmalltomediuminstallations,BINDisreallyonly
neededfortwothings:toactastheauthoritativesourceforadomainortwo
and provide forwarding to another DNS server for all other requests.
Here is a simple (but complete) named.conf that does exactly that:
options {
directory "/var/named";
pid-file "/var/run/named.pid";
statistics-file "/var/named/named.stats";
};
logging {
channel default_out {
file "/var/log/named.log";
};
category default { default_out; };
category config { default_out; };
category xfer-in { default_out; };
category xfer-out { default_out; };
category lame-servers { null; };
};166 | Information Servers
#78 Setting Up Caching DNS with Authority for Local Domains
HACK
zone "0.0.127.in-addr.arpa" in {
type master;
file "data/localhost.rev";
};
zone "." {
type hint;
file "rootservers.cache";
};
// Authoritative domains go here
zone "nocat.net" {
type master;
file "data/nocat.net";
};
Thismakesusauthoritativeforthedomainnocat.net,withitsdatastoredin
thefile /var/named/data/nocat.net.Requestsmadefordomainsotherthan
nocat.net(orfortheloopbacknetwork127.0.0.0)areautomaticallyfor-
wardedalongaccordingtotherootserverscontainedin rootservers.cache.A
suitablerootservercacheshouldhaveshippedwithBIND,butifyoucan’t
find it, try something like this:
# dig @a.root-servers.net > rootservers.cache
Ifyournetworkdoesn’thaveunrestrictedaccesstotheInternet(e.g.,you
arebehindarestrictivefirewall),thenforwardingtoauthoritativeservers
maynotwork.Ifthat’sthecase,tryanexplicitforwardingruleinplaceof
the hint entry above:
zone "." {
type forward;
forward only;
forwarders { 192.168.1.1; };
}
Naturally,replace 192.168.1.1 withtheIPaddressofavalidnameserverfor
yournetwork.ThiswilldirectallDNStraffictoyournetwork’sDNSserver,
whichpresumablyhaspermissiontododomainlookupsthroughthefirewall.
See also:
•“Views in BIND 9” [Hack #77]
•“Ultrahosting:MassWebSiteHostingwithWildcards,Proxy,and
Rewrite” [Hack #100]
• DNS & BIND, 4th Edition (O’Reilly)Distributing Server Load with Round-Robin DNS#79
Information Servers | 167
HACK
HACK
#79
Distributing Server Load with Round-Robin
DNS Hack #79
Direct traffic to multiple servers simultaneously with round-robin DNS
Ifyouserveaparticularlypopularsite,youwilleventuallyfindthewallat
whichyourserversimplycan’tserveanymorerequests.Inthewebserver
world,thisiscalledthe Slashdoteffect,anditisn’taprettysite(er,sight).
WhileaddingRAM,upgradingprocessors,andusingfasterdrivesandbuses
willhelpintheshortterm,youmayeventuallyfindthatasinglemachine
can’t possibly perform the work that needs to be done.
Onewaytoovercomethelimitsofthemonolithicserveristodistributethe
loadacrossmanymachines.Byaddingasecond(orthird)servertotheavail-
ablepoolofmachines,youcannotonlyincreaseperformancebutalsoadd
tothestabilityofthenetwork.Ifyouhaveahotspare(orthree)runningall
ofthetime,thenifonedevelopstrouble,theotherscantakeoverforitwith-
out any downtime.
Perhapstheeasiestwaytodistributetheloadofpublictrafficistohavethe
publicdotheworkofdistributingtheloadforyou.Throughthemagicof
round-robinDNS,inboundrequeststoasinglehostnamecanbedirectedto
come from any number of IP addresses.
InBIND9,thisisaseasyasaddingmultipleArecordsforasinglehost.For
example, suppose we use this in the zone file for oreillynet.com:
www 60 IN A 208.201.239.36
www 60 IN A 208.201.239.37
Now,whenahostslooksupwww.oreillynet.cominDNS,abouthalfofthe
time they will see:
rob@caligula:~$ host www.oreillynet.com
www.oreillynet.com has address 208.201.239.36
www.oreillynet.com has address 208.201.239.37
and the rest of the time, they get:
rob@caligula:~$ host www.oreillynet.com
www.oreillynet.com has address 208.201.239.37
www.oreillynet.com has address 208.201.239.36
AsmostapplicationsonlyusethefirstaddressreturnedbyDNS,thisworks
rathernicely.Approximatelyhalfoftherequestswillgotoeachaddress,and
thereforetheloadoftwoserversshouldberoughlyhalfofthatofasingle
server.WesettheTTLlow(to60seconds)topreventanyinterveningcach-
ingDNSserversfromhangingontoonesortorderfortoolong,whichwill
hopefully help keep the number of requests to each host more or less equal.168 | Information Servers
#80 Running Your Own Top-Level Domain
HACK
Itisonlyusefultospreadouttheloadifalloftheserversareinagreement
aboutwhatthey’reserving.Ifyourdatagetsoutofsync,thenbrowsers
mightgetoneversionofawebpageonthefirsthitandanotherwhenthey
hitreload.Ifyou’relookingforawaytokeepthatfromhappening,takea
look at“Keeping Parts of Filesystems in sync with rsync” [Hack #41].
Also,keepinmindthatdoingactualIPtakeover(incaseonehostisunable
toperformitsduties)istrickybusiness.Ifoneserverdies,youcan’tchange
DNSandwaitforittopropagatetotheentireInternet.You’llneedsome-
thingthattakesoverforthedownserverimmediately.Takealookatone
method of how to do this in“Cheap IP Takeover” [Hack #63].
See also:
•“Keeping Parts of Filesystems in sync with rsync” [Hack #41]
•“Cheap IP Takeover” [Hack #63]
HACK
#80
Running Your Own Top-Level Domain Hack #80
Set up your own TLD in BIND for ease of navigation
Ifyouadminanetworkthatusesprivateaddressing,you’vealmostcertainly
encounteredthedisassociatedschizophreniaoftryingtomaintainzonefiles
thatproperlyreflectinternalandexternalIPaddresses.Withtheintroduc-
tionofviewsin“ViewsinBIND9” [Hack#77],supportingmultipleaddress
ranges in a single domain has been significantly streamlined.
Whileusingviewsisonewaytoattacktheproblem,considertheeaseofset-
tingupyourowntop-leveldomain.Normally,zoneentriesin named.conf
look something like this:
zone "oreillynet.com" {
type master;
file "data/oreillynet.com";
};
ThisisanentryappropriateforanauthoritativeDNSserverfortheoreillynet.
comsubdomain.Theactualtopleveldomains(i.e.,.com,.net,.org,.int,etc.)
areonlydelegatedtothemysterious13knownastherootDNSservers.Even
thoughyourserverswon’tbeconsultedbytherestoftheInternet,itcanbe
handy to set up your very own TLD that works only on your local network.
Forexample,supposeyouhaveagroupofmachinesthatusetheprivate
192.168.1.0/24network.Thesemachinesaren’tdirectlyreachablefromthe
Internet,andyoudon’treallywanttoadvertisetheirDNSinformationto
would-be network crackers. Try a non-standard TLD:Monitoring MySQL Health with mtop#81
Information Servers | 169
HACK
zone "bp" {
type master;
file "data/bp";
allow-transfer { 192.168.1/24; };
allow-query { 192.168.1/24; };
};
(ThebpisshortforBackPlane,andmoretothepoint,isjustplainshort.)
Withtheaboveaddedtoyourzonefile,setupamasterrecordforbpjustas
you would any other domain:
$TTL 86400
@ IN SOA ns.bp. root.homer.bp. (
2002090100 ; Serial
10800 ; Refresh after 3 hours
3600 ; Retry after 1 hour
604800 ; Expire (1 week)
60 ; Negative expiry time
)
IN NS ns.bp.
ns IN A 192.168.1.1
homer IN A 192.168.1.10
bart IN A 192.168.1.11
lisa IN A 192.168.1.12
Reloadnamed,andyoushouldbeabletosimply pinghomer.bp.Ifyou’d
likeothernameserverstomaintainslavecopiesofyourTLD,justaddthem
as usual:
zone "bp" {
type slave;
file "db.bp";
masters { 192.168.1.1; };
};
Inthisway,youcanextendyournewTLDacrossyourentireprivatenet-
workarchitecture.Ifyou’rerunningtunnelsovertheInternet(asin“Tun-
neling:IPIPEncapsulation” [Hack#50])toconnectremoteofficesorfriends,
support for your TLD could theoretically grow to be as large as you like.
HACK
#81
Monitoring MySQL Health with mtop Hack #81
Display MySQL threads in real time in a format similar to top
Muchlikeits top counterpart,the mtop utilitygivesrealtime,runningstatis-
ticsofyourmysqlserverallinaterminalwindow.Onabusydatabase
server,thiscangiveyouveryprecisedetailsaboutwhatqueriesarerunning
(and taking up all of your resources).170 | Information Servers
#81 Monitoring MySQL Health with mtop
HACK
Whenrunning mtop,you’llneedtopassatleastthefollowingtwoswitches
on the command line:
mysql --dbuser=monitor --password=n0telling
Naturally,substitutingyourowndatabaseusernameandpassword.Ifyou’re
running mtop fromsomehostotherthanyourdatabaseserver,alsospecify
the --host={mysql_host} switch.Onceit’srunning,you’llbepresentedwith
a top-like screen that refreshes every few seconds:
load average: 0.72, 0.47, 0.26 mysqld 3.23.51 up 33 day(s), 4:48 hrs
2 threads: 2 running, 0 cached. Queries/slow: 71.5K/0 Cache Hit: 99.99%
Opened tables: 42 RRN: 4.0M TLW: 0 SFJ: 0 SMP: 0
ID USER HOST DB TIME COMMAND STATE INFO
26049 root localhost test Query show full processlist
26412 root localhost nocat 1 Query Writing to n select * from Member where
User like '%rob%'
---
Hereweseetheattachedusers,thehoststheyareconnectingfrom,theque-
riesthey’rerunning(andonwhichdatabases),aswellashowlongeach
threadhasbeenexecuting(inseconds).Thenumbersontheleftarethe
threadIDofeachrunningquery,notthemysqlPID.Ifaparticularqueryis
indangerofbecomingaslowquery,thelinewillturnmagenta.Ifthequery
thenbecomesaslowquery,thecolorchangestoyellow.Ifthethreadisstill
runningaftertwiceMySQL’slong_query_timevaluehaspassed,theline
becomesred.Thismakesiteasytotellataglanceifsomequeriesaretaking
a particularly long time to execute.
Justaswith top,hitting ? while mtop isrunningshowsallavailablekeystrokes:
mtop ver 0.6.2/2002905, Copyright (c) 2002, Marc Prewitt/Chelsea Networks
A top users display for mysql
These single-character commands are available:
q - quit
? - help; show this text
f - flush status
k - kill processes; send a kill to a list of ids
s - change the number of seconds to delay between updates
m - toggle manual refresh mode on/off
d - filter display with regular expression (user/host/db/command/state/info)
h - display process for only one host
u - display process for only one user
i - toggle all/non-Sleeping process display
o - reverse the sort order
e - explain a process; show query optimizer info
t - show mysqld stats (show status/mysqladmin ext)Monitoring MySQL Health with mtop#81
Information Servers | 171
HACK
v - show mysqld variables (show variables/mysqladmin vars)
z - zoom in on a process, show sql statement detail
r - show replication status for master/slaves
Probablythetwomostcommonlyusedfeaturesare explain (e)and kill (k).
HittingewillpromptyouforathreadID.Typeinthenumberofthequery
you’reinterestedin,anditwillshowyoudetailsaboutwhatmysqlisactu-
ally doing when running the query:
Id: 27134 User: root Host: localhost Db: nocat Time: 0
Command: Query State: cleaning up
select *
FROM Member
WHERE User like '%rob%'
table |type |possible_keys |key | ken_len|ref | rows|
Member |ALL | | | | | 9652|where used
Likewise,thekkeywillallowyoutokillathreadbysupplyingitsthreadID.
Thisistremendouslyhandyforkillinglong-running(orprocessintensive,
poorlyoptimized)queries,withouthavingtorun mysqladmin multipletimes
from the command line.
]Onesuggestedmethodforskippingtheneedforspecifyingtheuserand
passwordonthecommandlineistocreatea mysqltop userwithrestricted
privileges. From perldoc mtop:
Themostconvenientwaytosetupyoursystemtouse mtop istocreatea
databaseusercalledmysqltopwhichhasnopassword.Forsecuritypur-
poses,thisusershouldhaveallprivilegessettoNexcept Process_priv,which
must be set to Y.
Togranttheseprivileges,executethefollowingfromtheMySQLcommand
prompt:
mysql> grant select on test.* to mysqltop;
mysql> grant select on test.* to mysqltop@localhost;
mysql> update user set process_priv='y' where user='mysqltop';
mysql> flush privileges;
With mtop installedandinyourPATH,youcansaveyourselfalotoftyping
(andguesswork)comparedtohuntingdownmisbehavingthreadswith
mysqladmin alone.
See also:
•The mtop packageisavailableat http://mtop.sourceforge.net/ (which
requires Perl 5)
•Curses.pm (available through CPAN)172 | Information Servers
#82 Setting Up Replication in MySQL
HACK
HACK
#82
Setting Up Replication in MySQL Hack #82
Keeplivecopiesofyourdatabaserunningtoincreaseperformanceand
provide redundancy
AsofVersion3.23.33,databasereplicationisimplementedinMySQL.Itis
accomplishedbymaintainingabinarylogofeachdatabaseactiononone
machine(the master),andkeepingitinsyncwitheachofthereplicatedcop-
iesonothermachines(the slaves).AsofMySQL3.23,replicationisonly
one-way(thatis,youmaymakechangestothedatabaseonmasterthatget
propagatedtoeachslave,butchangestotheslavesarenotsentbacktothe
master).Ifyouneedbidirectionalreplication,takealookatthebleeding
edge code in MySQL 4.
Themostcommonreasonforsettingupreplicationistodistributetheload
ofhandlingdatabaserequestsacrossmultiplemachines.Thiswillnotonly
helpwithperformance,butwilladdabitofredundancyshouldoneofyour
databaseserversfail.Yourapplicationmustbesmartenoughtoknowthat
writerequests must behandedofftothemasterorelseyourreplicatedcop-
ieswillgetoutofsyncwiththemaster.Thisisnobody’sideaofafuntime,
somakesurethateveryplaceinyourcodethatneedstowritetothedata-
base only does so to the master copy.
HerearenineeasystepstogettingreplicationrunninginMySQL3.23.33(or
later).Beforestarting,makesurethatallofyourmachinesarerunningthe
same version of MySQL (preferably, the latest stable version).
1.Decidewhichdatabaseserverwillbethemaster,andwhichwillbethe
slave(s).Generally,themasterisyourmostcapablemachine.Ifyou
haveaparticularlypopularsitethatneedstohandlemanydatabase
requests,youshouldstronglyconsideramultiprocessormachinewith
plentyofRAMandahardwareraid.Ifyourdatabaseserverisunder-
powered,thenyourentiredynamicapplicationwillsufferasotherserv-
ersblockwhilewaitingforthedatabasetobeupdated,nomatterhow
many machines you distribute the load to.
2.Createareplicationuseronthemaster.ThisisaMySQLuserjustlike
anyother,onlywithFILEpermissionforeachdatabasethatyouwishto
replicate:
mysql> grant FILE on webdb.* to replicant@'%.mynetwork.edu' identified by
'seCret';
3.Enablethebinlogonthemaster,andpickwhichdatabaseswillberep-
licated.Includesomethinglikethisinthe[mysqld]sectionofthe /etc/
my.cnf on the master:
log-bin
server-id=1Setting Up Replication in MySQL#82
Information Servers | 173
HACK
Alsoincludebinlog-do-dblinesforeachdatabaseyou’dliketomake
available via replication.:
binlog-do-db=webdb
NowbringdownMySQLandimmediatelybringitbackupagainto
make your changes live:
root@db:/usr/local/mysql# mysqladmin shutdown; ./bin/safe_mysqld&
4.Bringdownthedatabaseonthemaster,andmakeanarchiveofthe
data/ directory.No,recentmysqldumpsarenotsufficientforthisjob;
youwillneedanactualcopyofthe data/ directory.Thismaytakeafew
secondstoseveralminutes,dependingonhowmuchdatayouhavein
your databases:
root@db:/usr/local/mysql# mysqladmin shutdown; tar cvf ~/data.tar data/
Yourdatabaseserverwillbeunavailableduringtheentiretimethatthe
copyisrunning,sobesuretorunthisatanoff-peaktime.Withany
luck, you’ll only have to do this once.
5.Bringthedatabasebackuponthemaster.Ifitlookslikeyour tar ran
okay, turn MySQL back on:
root@db:/usr/local/mysql# ./bin/safe_mysqld&
6.Copythearchivetoeachoftheslaves,andmakesureMySQLisn’trun-
ning on them. Try something like this on each slave:
root@slave:~# mysqladmin shutdown; scp db:data.tar .
7.PickauniqueIDforeachoftheslaves,andenablereplication.Include
thefollowinginthe[mysqld]sectionofthe /etc/my.cnf oneachofthe
slaves:
master-host=db.mynetwork.edu
master-user=replicant
master-password=seCret
server-id=10
Notethattheserver-id must beuniqueforeachslave.Chooseanyinte-
ger that hasn’t already been used for a slave (or the master).
Ifthisslavewillonlybeservingasaslavetothemaster-host(andwon’t
beservinganyread/writedatabasesofitsown)thenyoucangetabitof
a performance boost by including these options as well:
low-priority-updates
skip-bdb
delay-key-write-for-all-tables
Thiswilleliminatetheoverheadofsomeofthecodethatisonlyused
whenwritingtothedatabase.Ifyou’reactingasaslave,you’llnever
actuallywritetoadatabase,andsowon’tneedMySQLtobereadyto
handle it.
Download from Wow! eBook <www.wowebook.com>174 | Information Servers
#82 Setting Up Replication in MySQL
HACK
8.Extract the data archive on each slave. This should do it:
root@slave:/usr/local/mysql# mv data data.old; tar vxf ~/data.tar
Forgoodmeasure,makesurethatthenew data/ directoryanditscon-
tents are owned by the mysql user and group:
root@slave:/usr/local/mysql# chown -R mysql.mysql data/
Ifyoudon’tneedanythingfromtheoriginal data/ directoryontheslaves,
feel free to rm -rf them (rather than backing them up to data.old/).
9.StartMySQLoneachslave,andwatchtheerrorlogforproblems.
Finally, fire up MySQL on each slave:
root@slave:/usr/local/mysql# ./bin/safe_mysqld&
and watch the database log for errors:
root@slave:/usr/local/mysql# tail -f data/slave.err
(Naturally,substitutingyourmachine’srealnameforslave.)Ifyouseea
messagetotheeffectofStartingreplicationatpositionXYZ,thencon-
gratulations:you’rereplicating!Trymakinganupdatetothemaster,
andthendoaqueryonaslavetoseeifthechangepropagates.Updates
typicallyhappenalmostinstantaneously.Ifthereareanyproblemswith
replication,theywillbereportedinthemysqllogontheslavethat
encountered the problem.
Usually,problemswithreplicationtendtobepermissionproblems(check
yourGRANTsyntaxinstep2)orelseyourdatasetisoutofsynconthe
slaves(repeatsteps4,5,6,8,and9, very carefully.)Proceedslowly,and
always consult the mysql error logs on each machine.
Ifyouintendtoaddmoreslavesatalaterdate,thenkeepacopyofyour
data.tar handy.Youcanuseitatanytimetocreateanewslave(itwillsync
upwiththemasteroverthenetwork,replayingthebinlogthathasbeengen-
erated since the data.tar archive was created).
Simplyenablingreplicationandmaintainingareplicateddatabaseinstalla-
tionareverydifferentpropositions.Aslongasyourapplicationissmart
enoughtoonlymakeupdatestothemaster,andifyourhardwareisreli-
able,thenyougenerallywon’trunintodifficulty.Butshouldyourslavecop-
iesevergetoutofsyncforanyreason,thenyou’llneedtobearmedwith
muchmoreinformationthanIcanpresentinthishack.Consulttheexcel-
lentresourcesbelowbeforeattemptingtorunareplicatedinstallationofany
complexity.
See also:
• MySQL Reference Manual (O’Reilly)
•MySQLonlinedocumentation, http://www.mysql.com/doc/en/
Replication.htmlMySQL Server Tuning#84
Information Servers | 175
HACK
HACK
#83
Restoring a Single Table from a Large MySQL
Dump Hack #83
Here is a method for restoring a single mysql table from a huge mysqldump
Likeagoodadmin,youfaithfullydumpyour mysql tableseverynight,and
savethemtothefilesystemincompressedform(presumablytobepickedup
byabackupscriptlater).Youprobablyhavesomethinglikethisrunningin
cron on your database server (or one of its replicated slaves):
for x in `mysql -Bse show databases`; do
mysqldump $x | gzip -9 > /var/spool/mysqldump/$x.`date +%Y%m%d`.gz
done
Thiswillcoveryouifanythingcatastrophichappenstoyourlivedatabase.
Butifyourdatabasegrowstoanappreciablesize,doingpartialrestorescan
bedifficult.Onadatabasewithseveralmillionrows,yourdumpssuddenly
becomemassivepilesofdatathatneedtobesiftedthrough.Howcanyou
easilyrestoreasingletableoutofaseveralhundredmegabytecompressed
dump?
Here’sasimplemethodusingPerl.Createascriptcalled extract-table,with
this in it:
#!/usr/bin/perl -wn
BEGIN { $table = shift @ARGV }
print if /^create table $table\b/io .. /^create table (?!$table)\b/io;
ToextracttheUsertablefromthedumpofadatabasecalled randomdb,try
something like this:
# zcat /var/spool/mysqldump/randomdb.20020901.gz | extract-table Users > ~/
Users.dump
Now you can restore your Users table with a simple:
# mysql randomdb -e "drop table Users"
# mysql randomdb < ~/Users.dump
HACK
#84
MySQL Server Tuning Hack #84
TrythesepracticalstepstohelpmakeyourMySQLserverrunasefficiently
as it can
ManyLinuxadministratorsfindthemselvessuddenlythe“DBAinresi-
dence”whenthereisnobodyelsewilling(orable)totakeonthejob.Many
peoplespecializeintuningandmaintainingdatabasesforalivinganddon’t
touchsysadminresponsibilitiesatall,andyetmorethanoneLinuxadminis-
tratorI’veknownhasbeenrequiredtotakeonDBAresponsibilitieswithlit-
tletraining(andcertainlynoincreaseinpay).Whilethishackwon’tturn176 | Information Servers
#84 MySQL Server Tuning
HACK
youintoaDBAexpert,itwillhopefullyshowyousomepracticalstepsthat
have helped improve performance in real-world installations.
HerearefivestepsyoucantaketooptimizeyourMySQLinstallation,
roughly in increasing order of difficulty (and effectiveness).
1.Run mysqlcheck-odatabase.Thiswilloptimizeyourtables,reclaiming
lostspaceby“defragging”yourdatabase.Thisisespeciallyusefulifyou
haverecentlychangedthestructureofyourdatabase,orhavedeleteda
large amount of data from it.
2.Renice mysqld.IfyouarerunningadedicatedMySQLserver,youcan
telltheschedulertorunmysqlatamuchhigherprioritythanother
tasks.The mysql manualrecommendsaddingalinelikethistoyour
safe_mysqld script:
renice -20 $$
However,Ihavealsofounditnecessarytofindthefollowinghunkof
code:
NOHUP_NICENESS="nohup"
if test -w /
then
NOHUP_NICENESS=`nohup nice 2>&1`
if test $? -eq 0 && test x"$NOHUP_NICENESS" != x0 && nice --1 echo foo > /
dev/null 2>&1
then
NOHUP_NICENESS="nice --$NOHUP_NICENESS nohup"
else
NOHUP_NICENESS="nohup"
fi
fi
and replace it with simply:
NOHUP_NICENESS="nohup nice --20"
Now safe_mysqld andallofthemysqldprocesseswillrunatthegreat-
estpossiblepriority.Itdoesthisattheexpenseofallotherprocesses,so
ifyouaretryingtorunotherservicesalongwith mysql onthesame
machine,youmaywanttopickahighernumber(somewhereinthe-10
to -5 range is probably a bit more conservative).
3.Createindices.Ifyouhavelongrunningqueries,oneverygoodoptimi-
zationyoucanmakeistoaddappropriateindices.Ifyouseealongrun-
ningquerywhenrunning mtop (asdiscussedin“MonitoringMySQL
Health with mtop” [Hack #81]), then consider creating a relevant index:
mysql> create index name on Member (Name(10));
Indexingisatrade-offofdiskspaceforperformance,andinthisageof
inexpensivestorage,there’snoexcuseforneglectingyourindices.Hav-
ingtoomanyindicesusuallydoesn’thurt,butdoingalinearsearch(oraMySQL Server Tuning#84
Information Servers | 177
HACK
uniqueinsert)onalargetablethatdoesn’thaveanassociatedindexcan
make your server crawl.
4.Checkyourservervariables.Thedefaultservervariablesaredesignedto
providesafe,sensiblesettingsformodestlyequippedmachines.Ifyou
havealargeamountofRAM(512MBormore)thenyoucanseetremen-
dous benefits by increasing the size of the default buffers and caches.
Herearesomevariablesthatwerunonaproductiondatabaseserver(a
dualPentium4/1.0GHzwith2GBRAMandlotsoffastdiskspace).Put
them in the [mysqld] section of your /etc/my.cnf:
set-variable = key_buffer=384M
set-variable = max_allowed_packet=1M
set-variable = table_cache=512
set-variable = sort_buffer=2M
set-variable = record_buffer=2M
set-variable = myisam_sort_buffer_size=64M
set-variable = tmp_table_size=8M
set-variable = max_connections=768
Notethatmanyofthesearestraightoutofthe my-huge.cnf sample
includedwiththemysqldistribution(wehaven’thadtochangethem,
sincetheyworkfineforourinstallation.)SincewerunApache::DBIon
our web servers, we also run with a wait_timeout of a few minutes:
set-variable = wait_timeout=120
ThishelpspreventidleDBIthreadsfromhangingoutandtakingupall
available max_connections.
5.Patchglibcandthreads.Whenyouhaveexhaustedtheabilitiesofthe
defaultinstallationofglibc,considerpatchingit(see“Optimizingglibc,
linuxthreads,andtheKernelforaSuperMySQLServer” [Hack#86])to
allowforsmallerthreadsandmoreopenfiles.Thisisnormallynotan
issue except for very large, very busy MySQL installations.
Aswithmanytopicsinthisbook,databasetuningandadministrationisa
muchmorecomplicatedtopicthancanbecoveredinafewshortpages.Con-
sult the resources below for more authoritative discussion on the subject.
See also:
• MySQL Reference Manual (O’Reilly)
• MySQL (New Riders)
• http://www.mysql.com/doc/en/Linux.html
• http://www.mysql.com/doc/en/SHOW_VARIABLES.html178 | Information Servers
#85 Using proftpd with a mysql Authentication Source
HACK
HACK
#85
Using proftpd with a mysql Authentication
Source Hack #85
Eliminate the need for user accounts for ftp users with proftpd and mysql
The proftpd ftpdaemonisapowerfulftpdaemonwithaconfigurationsyn-
taxmuchlikeApache.Ithasawholeslewofoptionsnotavailableinmost
ftpdaemons,includingratios,virtualhosting,andamodularizeddesignthat
allows people to write their own modules.
Onesuchmoduleis mod_sql,whichallows proftpd touseaSQLdatabaseas
itsbackendauthenticationsource.Currently, mod_sql supportsMySQL
andPostgreSQL.Thiscanbeagoodwaytohelplockdownaccesstoyour
server,asinbounduserswillauthenticateagainstthedatabase(andthere-
forenotrequireanactualshellaccountontheserver).Inthishack,we’llget
proftpd authenticating against a MySQL database.
First, download and build the source to proftpd and mod_sql:
~$ bzcat proftpd-1.2.6.tar.bz2 | tar xf -
~/proftpd-1.2.6/contrib$ tar zvxf ../../mod_sql-4.08.tar.gz
~/proftpd-1.2.6/contrib$ cd ..
~/proftpd-1.2.6$ ./configure --with-modules=mod_sql:mod_sql_mysql \
--with-includes=/usr/local/mysql/include/ \
--with-libraries=/usr/local/mysql/lib/
(Naturally,substitutethepathtoyourmysqlinstall,ifitisn’tin /usr/local/
mysql/.) Now, build the code and install it:
rob@catlin:~/proftpd-1.2.6$ make && sudo make install
Next,createadatabasefor proftpd touse(assumingthatyoualreadyhave
mysql up and running):
$ mysqladmin create proftpd
then permit read-only access to it from proftpd:
$ mysql -e "grant select on proftpd.* to proftpd@localhost \
identified by 'secret';"
Create two tables in the database, with this schema:
CREATE TABLE users (
userid varchar(30) NOT NULL default '',
password varchar(30) NOT NULL default '',
uid int(11) default NULL,
gid int(11) default NULL,
homedir varchar(255) default NULL,
shell varchar(255) default NULL,
UNIQUE KEY uid (uid),
UNIQUE KEY userid (userid)
) TYPE=MyISAM;Using proftpd with a mysql Authentication Source#85
Information Servers | 179
HACK
CREATE TABLE groups (
groupname varchar(30) NOT NULL default '',
gid int(11) NOT NULL default '0',
members varchar(255) default NULL
) TYPE=MyISAM;
Onequickwaytocreatethetablesistosavetheabovetoafilecalled
proftpd.schema and run a command like mysql proftpd < proftpd.schema.
Nowweneedtotellproftpdtousethisdatabaseforauthentication.Addthe
following lines to /usr/local/etc/proftpd.conf:
SQLConnectInfo proftpd proftpd secret
SQLAuthTypes crypt backend
SQLMinUserGID 111
SQLMinUserUID 111
The SQLConnectInfo linetakestheformdatabaseuserpassword.Youcould
alsospecifyadatabaseonanotherhost(evenonanotherport)withsome-
thing like:
SQLConnectInfo proftpd@dbhost:5678 somebody somepassword
The SQLAuthTypes lineletsyoucreateuserswithpasswordsstoredinthe
standardUnixcryptformat,or mysql’sPASSWORD()function.Bewarned
thatifyou’reusing mod_sql’sloggingfacilities,thatthepasswordmaybe
exposed in plain text, so keep those logs private.
The SQLAuthTypes lineasspecifiedwon’tallowblankpasswords;ifyouneed
thatfunctionality,alsoincludetheemptykeyword.The SQLMinUserGID and
SQLMinUserUID specifytheminimumgroupanduseridthat proftpd willper-
mitonlogin.It’sagoodideatomakethisgreaterthanzero(toprohibitroot
logins)butshouldbeaslowasyouneedtoallowproperpermissionsinthe
filesystem.Onthissystem,wehaveauserandgroupcalledwww,withboth
itsuidandgidsetto111.Aswe’llwantwebdeveloperstobeabletologin
with these permissions, we’ll need to set the minimum values to 111.
Finally,we’rereadytocreateusersinthedatabase.Thiswillcreatetheuser
jimbo,witheffectiveuserrightsaswww/www,anddumphiminthe /usr/
local/apache/htdocs/ directory at login:
mysql -e "insert into users values ('jimbo',PASSWORD('sHHH'),'111', \
'111', '/usr/local/apache/htdocs','/bin/bash');" proftpd
Thepasswordforjimboisencryptedwith mysql’sPASSWORD()function
beforebeingstored.The /bin/bash lineispassedto proftpd topass proftpd’s
RequireValidShelldirective.Ithasnobearingongrantingactualshellaccess
to the user jimbo.180 | Information Servers
#86 Optimizing glibc, linuxthreads, and the Kernel for a Super MySQL Server
HACK
Atthispoint,youshouldbeabletofireup proftpd andloginasuserjimbo,
withapasswordofsHHH.Ifyouarehavingtroublegettingconnected,try
running proftpd in the foreground with debugging on, like this:
# proftpd -n -d 5
Watchthemessagesasyouattempttoconnect,andyoushouldbeableto
trackdownthesourceofdifficulty.Inmyexperience,it’salmostalwaysdue
toafailuretosetsomethingproperlyin proftpd.conf,usuallyregardingper-
missions.
The mod_sql modulecandofarmorethanI’veshownhere;itcanconnect
toexistingmysqldatabaseswitharbitrarytablenames,logallactivitytothe
database,modifyitsuserlookupswithanarbitraryWHEREclause,and
much more.
See also:
•Themod_sqlhomeisat http://www.lastditcheffort.org/~aah/proftpd/
mod_sql/
•The proftpd home is at http://www.proftpd.org/
HACK
#86
Optimizing glibc, linuxthreads, and the Kernel
for a Super MySQL Server Hack #86
Makesurethatyourdatabasesystem’sOSisrunningasefficientlyas
possible with these tweaks
IfyouhaveaverybusyMySQLserver(serving,say,800+queries/second
andhundredsofsimultaneousclients),youmaybegintorunintolimita-
tionsoftheoperatingsystemthatprevent mysql fromoperatingaseffi-
cientlyasitcould.Inextremecases(onsystemswithamisbehavingthreads
library), mysql hasbeenobservedtosuddenlytakeupallavailableCPU
cycles,artificiallydrivingtheloadto100+onceacriticalresourcethreshold
has been reached.
Bybuildingadedicated mysql server,andadjustingsomedefaultvaluesin
glibc,linuxthreads,andtheLinuxkernel,itispossibletomakeasingledata-
basemachineservethousandsofsimultaneousrequests.Naturally,you’ll
needhardwarecapableofsupportingtheload,butwiththesemodifica-
tions,theOSandMYSQLbuildwillprobablynotbethelimitingperfor-
mance factor.
WARNING:thefollowinghackmakeschangestocritical,sensitiveareasof
yourserver.Don’tgomonkeyingwithyourlibcandkernelsourcelightly,and
eventhenonlyifyouhaveacomplete,verified,offline(andpreferablyoff-site)
backupofyourentiresystem.ThisprocedureshouldonlybefollowedonOptimizing glibc, linuxthreads, and the Kernel for a Super MySQL Server#86
Information Servers | 181
HACK
dedicatedMySQLservermachines,andthenonlywhenyouarecertainthat
everythingelseisinorder(especiallyyour /etc/my.cnf variables).Youhave
been warned!
Step 1: Build glib. Downloadtheglibcsourcecodefrom http://www.gnu.org/
software/libc/libc.html.Thelatestversionasofthiswritingisglibc2.2.5.
Alsopickupacopyoflinuxthreadstogoalongwithwhicheverglibcyou
download.
Expandtheglibcarchive,andinstallthelinuxthreadsmodule(asperthe
instructionsintheglibcdistribution).Next,followthehackssuggestedat
http://www.mysql.com/doc/en/Linux.html, including:
• sysdeps/unix/sysv/linux/bits/local_lim.h setPTHREAD_THREADS_MAX
to 4096
• linuxthreads/internals.h set STACK_SIZE to 256 KB
Now,whenrunning configure,includethisswitch: --prefix=/usr/local/
glibc-2.2.5.Thiswillbuildglibcintoitsowndirectory,ratherthanreplace
thesystem’sexistingglibc.Thisiscriticalwhenrunning makeinstall,as
attemptingtooverwritearunningsystem’sglibcwilltypicallyresultin
librarysoup!Abrokenlibcisdifficulttofixwithoutstaticallybuiltsystem
toolsandabootdisk,andIdon’trecommendattemptingitunlessyouhave
timeonyourhands(andafullbackupofthemachineyoujustbroke).
Buildingglibctoanalternatedirectory(like /usr/local/glibc-2.2.5)neatly
sidesteps the problems of upgrading the system’s libc.
Nowbuildglibcandinstallit.Ifallwentwell,youshouldhaveashinynew
glibc installed to /usr/local/glibc-2.2.5/.
Step 2: The Kernel. Make the following changes to your kernel source tree:
• include/linux/limits.h set NR_OPEN to 4096
• include/linux/fs.h set INR_OPENto 4096
Now rebuild your kernel (and modules), install the new kernel, and reboot.
Step 3: Build a New MySQL. DownloadandunpacktheMySQLsourcetree.
Whenyourun configure,includethe --use-other-libc=/usr/local/glibc-
2.2.5 switch.Thiswilllinkmysqlagainstyournewglibcinsteadofthesys-
tem’s glibc. Now build and install MySQL as normal.
Step 4: Expand the Maximum Filehandles at Boot. Addthefollowinglineto
your /etc/rc.d/rc.local (orotherpartofthebootsequence,before safe_mysqld
runs):
echo 65536 > /proc/sys/fs/file-max182 | Information Servers
#87 Apache Toolbox
HACK
Nowreboot(orruntheabovecommandbyhand),andfireup safe_mysqld.
Inbenchmarkswehaverunin-house,themaximumnumberofavailable
connectionswasincreasedfromseveralhundredto4090,withnoobserved
CPUspike.Werundedicateddatabaseserversconfiguredwiththeseset-
tings in production, with absolutely no performance or stability issues.
Ifyouhaveasignificantamountofdatabasetraffic,youshouldconsiderdis-
tributingtheloadacrossmultiplemachinesdiscussedin“SettingUpRepli-
cation in MySQL” [Hack #82].
HACK
#87
Apache Toolbox Hack #87
Usethisgreatinstallationscripttoautomaticallydownload,configure,
compile, and install Apache (and friends)
BryanAndrew’sApacheToolboxisaSwissarmyknifeofascript,providing
acustomisable,menu-driveninterfacetodownloadingandcompiling
Apache, mod_perl,MySQL,PHP—andmore!TheApacheToolboxhasout-
of-the-box support for:
Apache
Our favorite web server
SSL
Secure Sockets Layer for secure web server interactions
PHP, mod_perl, and mod_fastcgi
For speedy scripting language support
MySQL
The ubiquitous lightning fast database server
OpenLDAP, mod_auth_ldap, mod_auth_radius, mod_auth_pop3, mod_auth_
sys and mod_accessref
Various means of authentication and authorization
WebDAV and mod_layout
For simple, yet powerful Web design functionality
mod_dynvhost, mod_throttle, mod_gzip, and mod_bandwidth
For efficient hosting and server control
Andmore!ApacheToolboxiscustomisableandhassupport(it’sjustashell
script after all) for anything else you’d care to plug in.
ApacheToolboxcomesintwodifferentflavours:justthescript,whichwill
downloadthevariouscomponentsourcesasneeded,orthefullpackage,
includingthescriptandallrequiredsources.ThetoolboxevencatchesRPM
conflicts should they arise.Apache Toolbox#87
Information Servers | 183
HACK
TheApacheToolboxitselfissimplyashellscriptthatrunsfromthecom-
mandline.Itmustberunasroot(asit’llremindyou)toinstallallthevari-
ousbitsintotheirappropriatelocations.Thefirststepisamenu-driven
GUIsh interface to selecting the various packages you wish to install:
-----------------------------------------------------------------------
Apache Toolbox 1.5.59
Support: http://www.apachetoolbox.com
-----------------------------------------------------------------------
[+] apache) Apache submenu...
[-] php) PHP submenu (v4.2.2)...
[-] rpm) Build an RPM with your choices?
[-] page2) Apache Modules PAGE 2 ...
-----------------------------------------------------------------------
[-] 1) GD 2.0.1 [-] 2) -SQL DB Menus-
[-] 4) Mod Python 2.7.8 [-] 5) Mod_SSL+OpenSSL
[-] 6) -Mod Throttle 312 [-] 7) -WebDAV 1.0.3-1.3.6
[-] 8) -Mod FastCGI [-] 9) -Mod AuthNDS 0.5
[-] 10) -Frontpage 2002 [-] 11) -Mod GZIP 1.3.19.1a
[-] 12) -Mod DynaVHost [-] 13) -Mod Roaming
[-] 14) -Mod AccessRef 1.0.1 [-] 15) -Mod AuthSYS
[-] 16) -Mod Bandwidth [-] 17) -Mod Perl 1.27
[-] 18) -Mod Auth LDAP [-] 19) -Apache Jakarta
[-] 20) -Mod Auth Radius [-] 21) -Mod Auth POP3
[-] 22) -Mod Layout 3.2 [-] 23) -Mod DTCL
q) Quit 99) Descriptions
go) Compile selections...
-----------------------------------------------------------------------
Notsurewhatallthesegoodiesare?Type 99<enter> forcompletedescrip-
tions.Afterputtingtogetheryourgrocerylistofpreferredfeatures,type go
and the toolbox springs into action. First stop: RPM conflict checks:
-------------------------------------------------------
-------------- Scanning for RPM's ---------------------
-------------------------------------------------------
Testing for PHP RPM... not found.
Testing for PHP IMAP RPM... not found.
Testing for GD RPM... not found.
Testing for GD Devel RPM... not found.
Testing for Apache RPM... not found.
...
Testing for OpenLDAP RPM... not found.
Testing for OpenLDAP Devel RPM... not found.
[+] Wget found!
-------------------------------------------------------
ThetoolboxconfirmsApacheinstallpath(/usr/local/apache),allowingyou
tochangethefinaldestinationperyourpreference,thencontinuesonits
way.
Download from Wow! eBook <www.wowebook.com>184 | Information Servers
#87 Apache Toolbox
HACK
Anytimeasource(tar.gz archive)isn’tfound,thetoolboxscriptprompts
you for permission to download it:
[+] Setting up Apache source...
[-] apache_1.3.26.tar.gz detection failed
Do you wish to download it now? [y/n] y
--21:35:40--
--ftp://mirrors.partnersforever.net:21/pub/apache/dist/
apache_1.3.26.tar.gz => `apache_1.3.26.tar.gz'
Connecting to mirrors.partnersforever.net:21... connected!
Logging in as anonymous ... Logged in!
==> TYPE I ... done. ==> CWD pub/apache/dist ... done.
==> PORT ... done. ==> RETR apache_1.3.26.tar.gz ... done.
Length: 2,303,147 (unauthoritative)
0K -> .......... .......... .......... .......... [ 2%]
50K -> .......... .......... .......... .......... [ 5%]
...
Thetoolboxthensilentlyhumsawayconfiguring,integrating,preheating,
and so on:
[+] Uncompressed Apache source...
[+] Getting apache pre-configured
[+] Apache pre-configured
[+] Apache httpd.conf-dist updated for SSI support
[+] Getting GD lib's with PNG and zlib support ready...
...
Afterashortwhile,thetoolboxgivesyouthechancetoedittheApachecon-
figurationscript,ifyou’rejustthatbold.Ifall’swellyoushouldendupwith
a rather familiar site if you’ve ever configured Apache before:
Configuring for Apache, Version 1.3.26
+ using installation path layout: Apache (config.layout)
+ activated php4 module (modules/php4/libphp4.a)
+ Warning: You have enabled the suEXEC feature. Be aware
+ that you need root privileges to complete the final
+ installation step.
Creating Makefile
Creating Configuration.apaci in src
Creating Makefile in src
+ configured for Linux platform
...
Creating Makefile in src/modules/standard
Creating Makefile in src/modules/php4
[+] Done Configuring Apache source
----------------------------------------------------------
If there where _no_ errors run "cd apache_1.3.26;make" now.
Start debugging and have a blast...
Run "make install" in the apache source
directory to install apache 1.3.26
----------------------------------------------------------Display the Full Filename in Indexes#88
Information Servers | 185
HACK
MoseyonovertotheApachedirectory,asspecified,make,maketest,and
make install!
See Also:
•Seetheoriginalarticleat http://www.onlamp.com/pub/a/apache/2000/11/
17/wrangler.html
•Apache Toolbox
• Linux Apache MySQL PHP (LAMP) Guide (Linux Help.Net)
• HTTP Wrangler columns (O’Reilly Network)
HACK
#88
Display the Full Filename in Indexes Hack #88
Stop truncating the filenames in your auto-indexed directories
HaveyouevernoticedthatApachewilltruncatefilenamesindirectorylist-
ings?It’sterriblyfrustratingtogotoawebpagefulloffunarchives,onlyto
see something like:
Index of /~rob/stuff/kernel
Name Last modified Size Description
Parent Directory 03-Sep-2002 00:33 -
patched-linux-2.4.12..> 11-Oct-2001 00:59 22.0M
patched-linux-2.4.12..> 11-Oct-2001 00:59 1k
patched-linux-2.4.12..> 11-Oct-2001 00:59 27.1M
patched-linux-2.4.12..> 11-Oct-2001 00:59 1k
patched-linux-2.4.13..> 23-Oct-2001 22:28 22.0M
patched-linux-2.4.13..> 23-Oct-2001 22:28 1k
patched-linux-2.4.13..> 23-Oct-2001 22:28 27.2M
patched-linux-2.4.13..> 23-Oct-2001 22:28 1k
patched-linux-2.4.14..> 05-Nov-2001 15:30 22.1M
patched-linux-2.4.14..> 05-Nov-2001 15:30 1k
patched-linux-2.4.14..> 05-Nov-2001 15:30 27.4M
patched-linux-2.4.14..> 05-Nov-2001 15:30 1k
patched-linux-2.4.15..> 22-Nov-2001 22:18 22.6M
patched-linux-2.4.15..> 22-Nov-2001 22:18 1k
patched-linux-2.4.15..> 22-Nov-2001 22:18 28.0M
patched-linux-2.4.15..> 22-Nov-2001 22:18 1k
Howcanyoutellwhichfilesarewhichwithouthoveringovereverylink?Fix
itbysettingyourIndexOptionslineinthe httpd.conf toincludeaName-
Width option:
IndexOptions FancyIndexing NameWidth=*
Bydefault,ApacheonlyshipswithFancyIndexingon.Nowissuean
apachectl restart, and reload that page to see the full filename:186 | Information Servers
#89 Quick Configuration Changes with IfDefine
HACK
Index of /~rob/stuff/kernel
Parent Directory 03-Sep-2002 00:33 - Description
patched-linux-2.4.12.tar.bz2 11-Oct-2001 00:59 22.0M
patched-linux-2.4.12.tar.bz2.sign 11-Oct-2001 00:59 1k
patched-linux-2.4.12.tar.gz 11-Oct-2001 00:59 27.1M
patched-linux-2.4.12.tar.gz.sign 11-Oct-2001 00:59 1k
patched-linux-2.4.13.tar.bz2 23-Oct-2001 22:28 22.0M
patched-linux-2.4.13.tar.bz2.sign 23-Oct-2001 22:28 1k
patched-linux-2.4.13.tar.gz 23-Oct-2001 22:28 27.2M
patched-linux-2.4.13.tar.gz.sign 23-Oct-2001 22:28 1k
patched-linux-2.4.14.tar.bz2 05-Nov-2001 15:30 22.1M
patched-linux-2.4.14.tar.bz2.sign 05-Nov-2001 15:30 1k
patched-linux-2.4.14.tar.gz 05-Nov-2001 15:30 27.4M
patched-linux-2.4.14.tar.gz.sign 05-Nov-2001 15:30 1k
patched-linux-2.4.15.tar.bz2 22-Nov-2001 22:18 22.6M
patched-linux-2.4.15.tar.bz2.sign 22-Nov-2001 22:18 1k
patched-linux-2.4.15.tar.gz 22-Nov-2001 22:18 28.0M
patched-linux-2.4.15.tar.gz.sign 22-Nov-2001 22:18 1k
Muchbetter.InstallthisbydefaultonallofyourApacheinstallations,and
your users will thank you.
HACK
#89
Quick Conﬁguration Changes with IfDeﬁne Hack #89
MakechangestoyourrunningApacheconfigurationwithouteditinghttpd.
conf
Apacheallowsyoutomodifyitsconfigurationquicklyandeasilywiththe
IfDefineandIfModuledirectives.IfDefineallowsyoutodesignateblocksof
configurationtoonlybeactivewhenacommandlineflaghasbeenpassed,
like so:
# /usr/local/apache/bin/httpd -DSSL
Theaboveisacommonexample,defining(-D)aflagcalledSSL.Thiswill
enable parts of the configuration that look like:
<IfDefine SSL>
# anything in here will be enabled if -DSSL
# has been passed on the command line.
</IfDefine>
Likewise,theIfModuledirectivecheckstoseeifamoduleiscurrentlyloaded
(i.e.,uncommented)intheApacheconfigurationfile.If mod_userdir.c is
loaded,forexample,thenthefollowingconfigurationwillbeenabled.Ifit’s
not, it will be silently ignored.
<IfModule mod_userdir.c>
UserDir public_html
</IfModule>Quick Configuration Changes with IfDefine#89
Information Servers | 187
HACK
AsApachehasgrown,the httpd.conf filehasbecomesteadilymoremodular-
izedasshownabove,allowingyoutotweakthevariousmodulesthatload
without fear of breaking your installation due to missing dependencies.
Withtheaboveexamplesinmind,youcaneasilychangehowApacheoper-
atesbypassingsimplecommandlines.Let’sassumethateveryFridayyou
releaseanewslettertotenthousandpeople.Youknowfromexperiencethat
theweekendsarewhenyourwebservergetshittheheaviest.Here’sasnip-
pet from a Apache configuration file using IfDefines:
<IfDefine !WEEKEND>
MinSpareServers 1
MaxSpareServers 5
StartServers 1
MaxClients 150
</IfDefine>
<IfDefine WEEKEND>
MinSpareServers 5
MaxSpareServers 15
StartServers 5
MaxClients 300
</IfDefine>
With the above configuration, if you start up your server normally:
# /usr/local/apache/bin/httpd
thenyou’llusethedefaultApachesettingsforthenumberofserverstostart
andmaintainduringnormalrunningoperation.However,beforeyouleave
work on Friday, you can simply stop the daemon and restart it with:
# /usr/sbin/httpd -DWEEKEND
Uponrestart,you’llhaveaconfigurationthat’sbettertunedfortheweek-
end traffic.
Here’s an example of one way to use IfModule:
<IfModule libperl.so>
PerlModule Apache::Registry
<Location /usr/local/httpd/cgi-bin>
SetHandler perl-script
PerlHandler Apache::Registry
Options +ExecCGI
</Location>
</IfModule>
Inthiscase, mod_perl isactivatedforallofour(hopefullywell-written)CGI
scriptsautomatically,wheneverthe mod_perl moduleisloaded.Thiscould
alsobeusefulduringthescenariodescribedabove—byturningon mod_perl,
ourscriptswillrespondfasterduringtheheaviertraffic.Ifwedon’tneedthe188 | Information Servers
#90 Simplistic Ad Referral Tracking
HACK
overheadof mod_perl duringtheweekdays(ortomakedebuggingcgiprob-
lemseasier),wecansimplycomment mod_perl’sLoadModuleand
AddModule lines, and we’ll be back to normal.
Youcanusetheideaspresentedabovetocreatedeveloperspecificconfigura-
tions,usablewitha-DEVELOPERcommand,orevena-DVIRTUAL_HOSTS
flag, which can be removed when your web hosting clients haven’t paid:
<IfDefine VIRTUAL_HOSTS>
# all your various <VirtualHosts> blocks would
# go in here. Without the command line flag of
# -DVIRTUAL_HOSTS (to signify that your clients
# have paid you, for instance), then all virtual
# hosts would be disabled.
</IfDefine>
<IfDefine EVELOPER>
# notice our trickery with "EVELOPER"
# to make the command line more readable.
# when -DEVELOPER has been passed, we load
# various modules that aren't used during production.
LoadModule proxy_module libexec/httpd/libproxy.so
LoadModule expires_module libexec/httpd/mod_expires.so
LoadModule usertrack_module libexec/httpd/mod_usertrack.so
AddModule mod_proxy.c
AddModule mod_expires.c
AddModule mod_usertrack.c
</IfDefine>
Ifyoursiteuses apachectl tobringApacheupanddown,youcanedit
apachectl toincludeyourfavoritedefines(it’sjustashellscript.)Findthis
line in apachectl:
HTTPD=/usr/local/apache/bin/httpd
and change it to something like this:
HTTPD="/usr/local/apache/bin/httpd -DSSL -DEVELOPER -DVIRTUAL_HOSTS"
Nowwhenyourun apachectlstart,your -D defineswillbeproperly
passed to httpd.
HACK
#90
Simplistic Ad Referral Tracking Hack #90
Create simple user tracking in print ads using the QUERY_STRING
You’replanningonadvertisinginanumberofmagazines,websites,and
newspapers,andyourealizethatyou’dliketogaugehowmuchtrafficis
cominginfromeachadvertisement—it’llhelpyoubetterspendyourmoney
inthefuture.Unfortunately,youhavenotimetoimplementsomething
complicated,soyou’relookingforalow-techsolution,aswellassomething
that will make it easy to analyze the results.Simplistic Ad Referral Tracking#90
Information Servers | 189
HACK
Apachehappilylogsallenvironmentvariablespassedtothepagesitserves
totheoutsideworld.Thus,thehackissimple:passanadvertisement-based
QUERY_STRINGtothemainpageofyoursite.TheQUERY_STRINGwill
beignoredifyoudon’tspecificallyactonit(withSSI,PHP,CGI,etc.),but
Apache will still log the information to it’s access_log.
Forexample,sayyou’readvertisingintheNewYorkTimes.Insteadofsay-
ing“Comevisitusat http://www.GatesMcFaddenCo.com,”changetheadver-
tisement to specifically match the readership:
http://www.GatesMcFaddenCo.com/?nyt
Intheaboveexample,anytimesomeonetypesinthataddress,Apachewill
servethemainpagenormallybutsilentlylogthataQUERY_STRINGof
“nyt”hasbeenpassed.Usingaloganalyzer(likeanalog,orthesimpleone
below),youcanthenfindouthowmanypeoplevisitedyoursitefromsee-
ing your ad in the New York Times.
Sinceyou’reusingQUERY_STRINGs,youcanalsoactuponthemwith
ServerSideIncludes,PHP,oranyotherserver-sidelanguage.Forinstance,
theexampleHTMLpagebelowwouldshowdifferentgreetingsbasedonthe
web site address that the user typed in.
<html>
<head>
<title>Apache Hack #12396 - Simplistic Ad Referrel Tracker</title>
</head>
<body>

<h1>Welcome New York Times Reader!</h1>

<h1>Welcome XML.com Reader!</h1>

<h1>Welcome To Our Site!</h1>

</body>
</html>
TheabovewillshowspecialgreetingsiftheusertypesinaQUERY_
STRINGthatcorrespondstoadvertisementsatXML.comorattheNew
YorkTimes.Iftheuserdidn’ttypeineither,thenagenericwelcomeis
shown.Somesiteshavebeenknowntochangemorethanjustthegreeting,
customizing the color, logos, and internal ads served.
Becarefulwhenyou’rechoosingyourQUERY_STRING—ifit’stoolong,
hardtoremember,orsuitablythreatening,thentheusermaymistypethe
address(creatingbadstatistics)orelsenottypetheQUERY_STRINGatall.
Inaworstcasescenario,theymaynotevenmaketheattempttovisit.These
are all bad choices:190 | Information Servers
#90 Simplistic Ad Referral Tracking
HACK
# this one is simply too long.
http://www.gamegrene.com/?newyorktimes-04-21
# this one would be hard to remember or type.
http://www.gamegrene.com/?04xlmfo3d
# and this may worry people.
http://www.gamegrene.com/?track=nyt
Ifyouareadvertisingonline,encodingquerystringsinanchortagsworks
justfine(andcanbeaslongasyoulike,asit’salljustoneclickasfarasthe
userknows).Toanalyzeyourstatsuponrequest,trythefollowingPerl
script.
Listing: referral-report.pl
#!/usr/bin/perl -w
use strict; my %ads;
# define each of your QUERY_STRINGS,
# and the matching "real name" here.
# this script will only look for
# QUERY_STRINGS that are letters and
# numbers only (no dashes, etc.)
$ads{"xml"} = "XML.com";
$ads{"nyt"} = "New York Times";
$ads{"ora"} = "O'Reilly and Associates";
# your Apache access log location.
my $logfile = "/var/log/httpd/access_log";
# define some counters.
my %ads_count; my $total;
# open the logfile.
open(LOG, "<$logfile") or die $!;
# and loop through each line.
while (<LOG>) {
# skip over lines we're not interested in.
next unless /GET/; next unless /\?(\w+)\s/;
# save the query_string.
my $query_string = $1;
# move on if not defined, else increment.
next unless exists $ads{$query_string};
$total++; $ads_count{$query_string}++;
}Mimicking FTP Servers with Apache#91
Information Servers | 191
HACK
# and print out the data.
print "There were a total of $total ad referrals.\n";
foreach ( sort keys %ads_count ) {
print "$ads{$_} had $ads_count{$_} ad referrals.\n";
}
# close logfile and
# exit the program.
close(LOG); exit;
The results of this script would look like:
There were a total of 17 ad referrals.
XML.com had 6 ad referrals.
New York Times had 7 ad referrals.
O'Reilly and Associates had 4 ad referrals.
Beingsimplistic,thiscodedoesnottrackmultiplehitsfromthesameIP
address,nordoesitcreatepercentagesbasedonthetotalamountofrefer-
rals received. Both should be easy for any Perl hacker to implement.
HACK
#91
Mimicking FTP Servers with Apache Hack #91
Set up multiple levels of anonymous access in Apache
YouwanttosetupaportionofyourwebsitetoactlikeanFTPserver—
someusersshouldhaveaccountsfordownloading,whileothersshouldhave
anonymousdownloadingaccess.Stillothersshouldbeallowedtodown-
loadonetypeoffilebutnotanother.Youdon’ttrusttheintelligenceofyour
clienteletohandleinstallingandusinganFTPprogram,soyou’dliketo
make things work within a normal browsing environment.
Usinganoft-ignoredApachemoduleincludedbydefault,youcanallow
authenticatedanonymousaccesstocertainpartsofyoursite,aswellasfalling
backonnormalauthenticationmethodswith mod_auth.Thisbecomeshandy
ifyou’vegotalargeamountofdocumentsthatyoudon’twantindexedby
searchengines,oryouwanttocreatetheeffectofmultiplelevelsofuser
power,suchasanonymous,privileged,andleech.Youwanttodoitall
through a browser and all without additional programming.
Thefirststepistoenablethe anon_auth_module,whichisnormallycom-
mentedoutwithintheApacheconfigurationfile.Todoso,lookforthefol-
lowinglines(whichmaybeslightlydifferentinyourinstallation)and
remove the #:
#LoadModule anon_auth_module libexec/httpd/mod_auth_anon.so
#AddModule mod_auth_anon.c
WiththeaboveuncommentedandApacherestarted,youcannowstartadd-
ingrelevantdirectivesto httpd.conf,oran .htaccess filelocatedinthedirectory
Download from Wow! eBook <www.wowebook.com>192 | Information Servers
#91 Mimicking FTP Servers with Apache
HACK
youwantprotected.Below,we’llassumeyou’reusingan .htaccess file.Plop
thefollowingintoyour .htaccess andsavetothedirectoryyouwanttoprotect:
AuthName "anonymous/your email address"
AuthType Basic
Require valid-user
Anonymous orko bender
Anonymous_Authoritative on
Youshouldrecognizethefirstthreedirectives,asthey’reusedwhenyou
normallyprotectadirectorywith mod_auth.TheAnonymousdirectivecon-
trolswhatusernamesshouldbeconsideredananonymoususer—inthis
case,we’vegotorkoandbender,butwecouldjusthaveeasilychosenmun-
dania such as anonymous, anon, or guest.
TheAnonymous_Authoritativecontrolswhetherwewanttopassunautho-
rizedusernamesandpasswordsofftoanotherauthenticationschemefor
processing.Ifwesayon,thenanonymityisking,eitherthevisitorlogsin
with orko or bender or they’re not allowed access.
Ontheotherhand,ifwesayoff,thenwecanincludethefullfunctionalityof
mod_auth—authenticationviapasswords,groups,andsoforth.Takealook
attheconfigurationbelow.Iftheuserdoesnotloginwithheenieorretro-
girl,thentheusernameispassedofftotheAuthUserFile,whereit’salso
checkedagainstforthoseaccounts.Iftheydon’texistinthatfileeither,then
the user is denied:
AuthName "anonymous/your email address"
AuthUserFile /Library/WebServer/.htpasswd
AuthType Basic
Require valid-user
Anonymous heenie retrogirl
Anonymous_Authoritative off
Asistypical,youcanbeassimplisticorascomplicatedasyouneed.Thefol-
lowingconfigurationwillallowanyusertogetadirectorylisting(allowing
themtoseewhatyouhaveavailable).AnyuserlistedintheAuthUserFile
cangetaccesstoall .jpg files,aswellasanyanonymoususerlogginginwith
themrs_decepticonorspiderjusernames,assumingtheyenteravalidemail
address(onewitha“@”and“.”character—Apachedoesn’tcheckthatthe
hostname exists). Finally, only the eustace user can download .mp3 files:
AuthType basic
AuthName "anonymous/your email address"
AuthUserFile /Library/WebServer/.htpasswd
Anonymous mrs_decepticon spiderj
Anonymous_Authoritative off
Anonymous_VerifyEmail onRotate and compress Apache Server Logs#92
Information Servers | 193
HACK
<Files *.jpg>
Require valid-user
</Files>
<Files *.mp3>
Require user eustace
<Files>
Youcan,ofcourse,getevenmoreconvoluted,restrictingbyIPaddressor
hostname,environmentvariables,andsoforth.Justmakesureyoucom-
mentyourcraziness—it’sveryeasytogetconfusedaboutwhohasaccessto
what.
Unfortunately,youcan’tduplicatetheothersideofFTPandthat’supload-
ing.NeithercananyofthemodulesthatareshippedwithApacheby
default.Todoso,you’dhavetouseoneofthemanyCGIscriptsthataccept
file uploads, or else craft your own.
HACK
#92
Rotate and compress Apache Server Logs Hack #92
UsethistinyPerlscripttocompressallofyourApachelogsautomatically,
even when you add more to your site
Here’sahandyscriptthatwillrotateandgzipyourApachelogsforyou
automatically.Itreadsyour httpd.conf,descendsallIncludedconfiguration
files,andmakesanoteofeachuniqueLogfile.Itthenrenameseachtoafile
withadatestamp,restartsApache,thengzipcompressesthedatestamped
logfiles.Asabonus,it chgrpsthecompressedlogstowhatevergroupyou
havesetin$gid,andsetspermissionsupforthatgrouptobeabletoreador
writetothem.Thismakesitsimpleforother(non-root)processestocome
along later and post-process the compressed logs.
Runthisnightly(orweekly,dependingonyourtraffic)in cron tocollecta
regularrepositoryofcompressedlogs,andkeepyourlivelogfilesdowntoa
manageable size.
Listing: logﬂume.pl
#!/usr/bin/perl -w
#
# logflume.pl
#
# Roll over and compress Apache log files, following Includes within the
# httpd.conf (and all other configuration files).
#
#
use strict;
$|++;194 | Information Servers
#93 Generating an SSL cert and Certificate Signing Request
HACK
my $server_root = "/usr/local/apache";
my $conf = "$server_root/conf/httpd.conf";
my $gid = "wwwadmin";
my (%logs, %included, @files, @gzip);
my $date = `date +%Y%m%d`; chomp $date;
push @files, $conf;
for $conf (@files) {
open(CONF, "<$conf") || die "Cannot open config file $conf: $!\n";
while (<CONF>) {
chomp;
next if /^(\s+)?#/;
if (/(Transfer|Custom|Error)Log\s+(\S+)(\s+)/i) {
$logs{$2}++;
} elsif (/^(ResourceConfig|Include)\s+(\S+)/i) {
if(!$included{$2}) {
push @files, $2;
$included{$2}++;
}
}
}
close CONF;
}
for my $logfile (sort keys %logs) {
$logfile = "$server_root/$logfile" unless ($logfile =~ m|^/|);
rename($logfile, "$logfile.$date");
push(@gzip, "$logfile.$date");
}
system("$server_root/bin/apachectl restart");
for my $logfile (@gzip) {
system("gzip $logfile");
system("chgrp $gid $logfile.gz");
system("chmod 664 $logfile.gz");
}
HACK
#93
GeneratinganSSLcertandCertiﬁcateSigning
Request Hack #93
Make an SSL key, CSR, and cert for use with Apache
InordertouseApachewith mod_ssl or Apache-ssl,you’llneedacertificate
signedbyatrustedCertificateAuthority.Inthisexample,we’llassumethatGenerating an SSL cert and Certificate Signing Request#93
Information Servers | 195
HACK
you’regeneratingacerttobeusedat https://propaganda.discordia.eris/.To
generate a key with OpenSSL:
hagbard@fnord:~/certs$ openssl genrsa 512/1024 \
> propaganda.discordia.eris.key
warning, not much extra random data, consider using the -rand option
Generating RSA private key, 512 bit long modulus
..++++++++++++
...++++++++++++
e is 65537 (0x10001)
Thisjustmakestheprivatekey,notthecert.Ifyou’dliketoprotectthiskey
with a passphrase, use the -des3 option on the command line:
hagbard@fnord:~/certs$ openssl genrsa -des3 512/1024 \
> propaganda.discordia.eris.key
warning, not much extra random data, consider using the -rand option
Generating RSA private key, 512 bit long modulus
.......++++++++++++
.....++++++++++++
e is 65537 (0x10001)
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:
Butbewarned:you’llneedtoenterthisphraseeverytimeyourestart
Apache,whichcanbeinconvenientwhenperformingregularmaintenance
(suchasrotatinghttplogs).Weightheinconvenienceagainstthepotential
damagedoneifsomemiscreantshouldacquirethiskey.Ifyoulosethepass-
phrase, it is essentially unrecoverable, so keep it safe!
Nextyou’llneedtogeneratetheCertificateSigningRequest,tosubmittoa
trustedCA(suchasThawte/VeriSign)forsigning.Typeineverythingappear-
ing in boldface, substituting your own information where appropriate:
hagbard@fnord:~/certs$ opensslreq-new-keypropaganda.discordia.eris.key\
> propaganda.discordia.eris.csr
Using configuration from /usr/local/ssl/openssl.cnf
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Texas
Locality Name (eg, city) []:Mad Dog
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Discordia, Inc.
Organizational Unit Name (eg, section) []:Operations
Common Name (eg, YOUR name) []:propaganda.discordia.eris
Email Address []:norton@discordia.eris196 | Information Servers
#94 Creating Your Own CA
HACK
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Finally,you’rereadytocreatethecertitself.Ifyou’regoingtouseawell-
knownCA,thenyou’llneedtosendthe .csr filetothemforsigning.Inthe
meantime,youcanself-signyourcertanduseitwhileyou’rewaitingforthe
CSR to be processed:
hagbard@fnord:~/certs$ openssl req -x509 \
-key propaganda.discordia.eris.key \
-in propaganda.discordia.eris.csr \
> propaganda.discordia.eris.crt
Using configuration from /usr/local/ssl/openssl.cnf
hagbard@fnord:~/certs$
IfyouintendtobuildyourownCertificateAuthority,then useyourownCA
key to sign your cert.
See also:
•“Creating Your Own CA” [Hack #94]
•The OpenSSL home at http://www.openssl.org/
HACK
#94
Creating Your Own CA Hack #94
BecomeyourownCertificateAuthority,andsignyourown(orothers’)SSL
certs
Well-knownCertificateAuthorities(suchasThawte/VeriSign)existtoserve
asanauthoritative,trustedthird-partyforauthentication.Theyareinthe
businessofsigningSSLcertificatesthatareusedonsitesthatdealwithsen-
sitiveinformation(suchasaccountnumbersorpasswords).Ifasite’sSSL
certificateissignedbyatrustedauthority,thenpresumablyitispossibleto
verifytheidentityofaserversupplyingthatcert’scredentials.Inorderto
receiveacertificate“blessed”byawellknownCA,youhavetoproveto
thembeyondashadowofdoubtthatnotonlyareyouwhoyouclaimtobe,
butthatyouhavetherighttousethecertificateinthewayyouintend.For
example,ImaybeabletoprovetoaCAthatIamreallyRobFlickenger,but
theyprobablywon’tissuemeasignedcertforMicrosoftCorporation,asI
havenorightstousethatname.Yes,theyprobablywouldn’tdothat.Not
again.
OpenSSLisperfectlycapableofgeneratingeverythingyouneedtorunyour
own Certificate Authority. The CA.pl utility makes the process very simple.Creating Your Own CA#94
Information Servers | 197
HACK
Intheseexamples,you’llneedtotypeanythinginboldface,andenterpass-
wordswhereverappropriate(thatdon’techotothescreen.)Toestablish
your new Certificate Authority:
hagbard@fnord:~/certs$ /usr/local/ssl/misc/CA.pl -newca
CA certificate filename (or enter to create)
Making CA certificate ...
Using configuration from /usr/local/ssl/openssl.cnf
Generating a 1024 bit RSA private key
...............++++++
......................................++++++
writing new private key to './demoCA/private/cakey.pem'
Enter PEM pass phrase:
Verifying password - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Locality Name (eg, city) []:Sebastopol
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Illuminatus
Enterprises, Ltd
Organizational Unit Name (eg, section) []:Administration
Common Name (eg, YOUR name) []:Hagbard Celine
Email Address []:hagbardceline1723@yahoo.com
Congratulations.You’retheproudownerofyourveryownCertificate
Authority. Take a look around:
hagbard@fnord:~/certs$ ls
demoCA/
hagbard@fnord:~/certs$ cd demoCA/
hagbard@fnord:~/certs/demoCA$ ls -l
total 24
-rw-r--r-- 1 rob users 1407 Sep 8 14:12 cacert.pem
drwxr-xr-x 2 rob users 4096 Sep 8 14:12 certs/
drwxr-xr-x 2 rob users 4096 Sep 8 14:12 crl/
-rw-r--r-- 1 rob users 0 Sep 8 14:12 index.txt
drwxr-xr-x 2 rob users 4096 Sep 8 14:12 newcerts/
drwxr-xr-x 2 rob users 4096 Sep 8 14:12 private/
-rw-r--r-- 1 rob users 3 Sep 8 14:12 serial
ThepublickeyforyournewCertificateAuthorityiscontainedincacert.
pem,andtheprivatekeyisinprivate/cakey.pem.Youcannowusethispri-
vate key to sign other SSL certs.198 | Information Servers
#94 Creating Your Own CA
HACK
TouseyourCA’sauthoritytosignSSLcerts,you’llneedtomakeanewcert
thatawebserver(suchasApache)canuse.First,generateaprivatekeyand
certificaterequest,asshownin“GeneratinganSSLcertandCertificateSigning
Request” [Hack#93].NowyoucansignthenewrequestwithyourownCA’skey:
hagbard@fnord:~/certs$ openssl ca -policy policy_anything \
-out propaganda.discordia.eris.crt \
-infiles propaganda.discordia.eris.csr
Using configuration from /usr/local/ssl/openssl.cnf
Enter PEM pass phrase:
Check that the request matches the signature
Signature ok
The Subjects Distinguished Name is as follows
countryName :PRINTABLE:'US'
stateOrProvinceName :PRINTABLE:'Texas'
localityName :PRINTABLE:'Mad Dog'
organizationName :PRINTABLE:'Discordia, Inc.'
organizationalUnitName:PRINTABLE:'Operations'
commonName :PRINTABLE:'propaganda.discordia.eris'
emailAddress :IA5STRING:'hail@discordia.eris'
Certificate is to be certified until Sep 8 22:49:26 2003 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Nowtousethe .crt and .key withApache+mod_ssl(or Apache-ssl),install
them as you normally would (perhaps with lines like these):
SSLCertificateFile /usr/local/apache/conf/ssl.crt/propaganda.discordia.eris.
crt
SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/propaganda.discordia.
eris.key
Thisisalllotsoffun,butwhathappenswhenaclientactuallyconnectsto
https://propaganda.discordia.eris/?Won’tthebrowserthrowanerrorabout
notrecognizingtheCertificateAuthoritythatsignedtheSSLcert?Natu-
rally.Unless,ofcourse,you’veinstalledyourCA’spublickeytotheclient
browser ahead of time. See the next hack if you’d like to do that.
See also:
•man CA.pl
•The OpenSSL home at http://www.openssl.org/
• http://www.cert.org/advisories/CA-2001-04.html
Disclaimer:no,Ihonestlyhad nothing todowiththeMicrosoftCorpora-
tioncertsnafu.Butitdoesillustrateoneofthefundamentalfactsoflife
online: it’s difficult to know who to trust.
Download from Wow! eBook <www.wowebook.com>Distributing Your CA to Client Browsers#95
Information Servers | 199
HACK
HACK
#95
Distributing Your CA to Client Browsers Hack #95
Installing your shiny new CA cert to client browsers is just a click away
Therearetwopossibleformatsthatbrowserswillacceptfornewcertificate
authoritycerts: pem and der.EarlyversionsofNetscapeexpected pem for-
mat,butrecentversionswillaccepteither.InternetExplorerisjustthe
opposite(earlyIEwouldonlyaccept der format,butrecentversionswill
takeboth).Otherbrowserswillgenerallyaccepteitherformat.Youcangen-
erate a der from your existing pem with a single openssl command:
hagbard@fnord:~/certs$ openssl x509 -in demoCA/cacert.pem \
-outform DER -out cacert.der
Also,addthefollowinglinetoyour conf/mime.types fileinyourApache
installation:
application/x-x509-ca-cert der pem crt
NowrestartApacheforthechangetotakeeffect.Youshouldnowbeableto
placeboththe cacert.der and demoCA/cacert.pem filesanywhereonyourweb
server, and have clients install the new cert by simply clicking on either link.
Youwillgetadialogboxinyourbrowserwhendownloadingthenewcertif-
icateauthority,askingifyou’dliketocontinue.Acceptthecertificate,and
that’sallthereistoit.NowSSLcertsthataresignedbyyourCAwillbe
accepted without warning the user, as inFigure8-1.
KeepinmindthatCertificateAuthoritiesaren’ttobetakenlightly.Ifyou
acceptanewCAinyourbrowser,youhadbettertrustitcompletely—amis-
chievousCAmanagercouldsignallsortsofcertsthatyoushouldnever
Figure8-1.Click OK to accept the new Certificate Authority or View to read the fine
print200 | Information Servers
#95 Distributing Your CA to Client Browsers
HACK
trust,butyourbrowserwouldnevercomplain(sinceyouclaimedtotrust
theCAwhenyouimportedit).Beverycarefulaboutwhoyouextendyour
trusttowhenusingSSLenabledbrowsers.It’sworthlookingaroundinthe
CAcachethatshipswithyourbrowsertoseeexactlywhoyoutrustby
default.
Forexample,didyouknowthatAOL/TimeWarnerhasitsownCA?How
aboutGTE?OrVISA?CAcertsforalloftheseentities(andmanyothers)
shipwithNetscape7.0forLinuxandarealltrustedauthoritiesforwebsites,
email,andapplicationadd-onsbydefault.Keepthisinmindwhenbrows-
ingtoSSL-enabledsites:ifanyoneofthedefaultauthoritieshavesigned
onlinecontent,thenyourbrowserwilltrustitwithoutrequiringoperator
acknowledgment, as inFigure8-2.
Ifyouvalueyourbrowser’ssecurity(and,byextension,thesecurityofyour
clientmachine),thenmakeitapointtoreviewyourtrustedCArelationships.
See also:
•The OpenSSL FAQ at http://www.openssl.org/support/faq.cgi
•“Generating an SSL cert and Certificate Signing Request” [Hack #93]
•“Creating Your Own CA” [Hack #94]
Figure8-2.It’s a good idea to review exactly who your browser considers trustworthy.Serving multiple sites with the same DocumentRoot#96
Information Servers | 201
HACK
HACK
#96
Serving multiple sites with the same
DocumentRoot Hack #96
Throughcreativeuseofmod_rewrite,severalsitescansharea
DocumentRoot and yet still appear to be independent sites
Occasionally,itisusefulforacoupleofsitestosharethesameDocumentRoot
buthaveuniquefrontpages.Perhapsyouwanttoshareagraphicsorjavas-
criptdirectory,butforsomereasoncan’tbebotheredtosetupanappropriate
Aliasentry.Ormaybeyouneedtobeabletosharemostofadirectorystruc-
ture,butdon’twanttoFollowSymLinks.Whateveryourreasonforneedinga
common DocumentRoot, mod_rewrite will provide a solution.
Let’sassumethatyouarerunningasitecalled www.fruit.yum,andwould
liketoserveinformationaboutapplesandoranges.You’vealreadyhad
muchsuccesswith http://www.fruit.yum/apples and http://www.fruit.yum/
oranges,butyouaresuddenlyabletoobtainthemuchcoveted apple.yum
and orange.yum domainnames.You’dliketokeeptheexisting www.fruit.
yum directorystructureforuserswhobrowsetodirectoriesunderyournew
domainnames(forexample,to http://www.apple.yum/order or http://www.
orange.yum/faq),butwouldliketodeliveracustompagetouserswhosim-
ply browse to the front page (e.g., http://www.apple.yum/).
Thisisnoproblemwithmod_rewrite.Let’sassumethatyoualreadyhavea
VirtualHost entry for www.fruit.yum like this:
<VirtualHost *>
ServerName www.fruit.yum
ServerAdmin webmaster@fruit.yum
DocumentRoot /home/www/htdocs
CustomLog /home/www/logs/access_log combined
ErrorLog /home/www/logs/error_log
</VirtualHost>
AddacoupleofnewVirtualHostentriesforyournewdomainsbutwith
some additional RewriteRule lines:
<VirtualHost *>
ServerName www.apples.yum
ServerAdmin webmaster@fruit.yum
DocumentRoot /home/www/htdocs
CustomLog /home/www/logs/access_log combined
ErrorLog /home/www/logs/error_log
RewriteEngine On202 | Information Servers
#96 Serving multiple sites with the same DocumentRoot
HACK
RewriteRule ^/$ /home/www/htdocs/apples/index.html
RewriteRule ^/index.html$ /home/www/htdocs/apples/index.html
</VirtualHost>
<VirtualHost *>
ServerName www.oranges.yum
ServerAdmin webmaster@fruit.yum
DocumentRoot /home/www/htdocs
CustomLog /home/www/logs/access_log combined
ErrorLog /home/www/logs/error_log
RewriteEngine On
RewriteRule ^/$ /home/www/htdocs/oranges/index.html
RewriteRule ^/index.html$ /home/www/htdocs/oranges/index.html
</VirtualHost>
Asthesearehandledasinternalrewrites,theURLlinewillnotchangeon
theclient’sbrowser.Asfarastheyareconcerned,browsingto http://www.
oranges.yum/ appearstodisplayanindependentsite.Browsingdirectlyto
http://www.fruit.yum/oranges/ isfunctionallyequivalent,aslongasrelative
links(orappropriateAliaslines)areusedinallofyourhtml.Ifyou’dlikea
moregeneralsolutionthatwillworkforasmanywww.*.yumaddressesas
you care to register, you might try something like this:
<VirtualHost *>
ServerName www.fruit.yum
ServerAlias www.*.yum
ServerAdmin webmaster@fruit.yum
DocumentRoot /home/www/htdocs
CustomLog /home/www/logs/access_log combined
ErrorLog /home/www/logs/error_log
RewriteCond %{HTTP_HOST} www.(.*).yum
RewriteCond %{HTTP_HOST} !www.fruit.yum
RewriteRule (.*) $1 [E=SITE:%1]
RewriteCond %{REQUEST_URI} ^/index.html$
RewriteCond /home/www/htdocs/%{ENV:SITE}/index.html -f
RewriteRule .* /home/www/htdocs/%{ENV:SITE}/index.html
RewriteCond %{REQUEST_URI} ^/$
RewriteCond /home/www/htdocs/%{ENV:SITE}/index.html -f
RewriteRule .* /home/www/htdocs/%{ENV:SITE}/index.html
</VirtualHost>Delivering Content Based on the Query String Using mod_rewrite#97
Information Servers | 203
HACK
Thiswillstripoffthemiddlepartoftherequesteddomainname(thebit
betweenwww.and.yum)andchecktoseeifan index.htmlexistsinthe
directoryofthatnameundertheDocumentRootforthesite.Ifso,andifthe
request was for / or /index.html, then it performs an internal rewrite to it.
Workingwith mod_rewrite canbeconfusing.Onetoolthathelpsgreatly
withdebuggingrewriteproblemsistheRewriteLogfunctionality.Ifyou’re
stuckonfiguringoutaRewriteRule,tryacoupleoflineslikethisinyour
VirtualHost entry:
RewriteLog /tmp/rewrite.log
RewriteLogLevel 9
Thiswillturnonfulldebuggingtothelogyouspecify.Itishandytowatchit
witha tail-f/tmp/rewrite.log or less-f/tmp/rewrite.log asyou
requestvariousURLsinabrowser.Don’trunwiththisenabledforverylong
onaproductionserver,asallrewriteactionswillbelogged(andconse-
quently make your Apache run slower).
HACK
#97
Delivering Content Based on the Query String
Using mod_rewrite Hack #97
Control content delivery based on a URL’s query string without a CGI script
ItcanbeusefultousethequerystringofaURLline(thatis,anythingafter
thefirst ?)todirectwhatcontentApachewillserve.Whilethequerystring
isnormallyusedbyCGIapplicationstoreadandwriteprogramvariables,
mod_rewrite can use it without the need for an external script.
Forexample,supposeyouhaveapublicationsystemthatsplitslargearti-
clesintomultiplepages.Thecontentisnormallydeliveredbyascriptthe
takesthepage=querystringanddeliverstherequestedpagenumber.Ifyou
cansavealocalcopyofeachpagetothefilesystem,thenyoucanusea
RewriteRuletodeliverthecachedcopy,andbypasstheoverheadofrunning
the script on every hit.
Let’ssupposewe’retryingtoservethesecondpageofanarticleat http://
mysite.com/news/article.html?page=2. Install the following rules:
RewriteCond %{QUERY_STRING} page=([0123456789]+)
RewriteCond /home/www/htdocs/%{REQUEST_URI}.%1 -f
RewriteRule .* /home/www/htdocs/%{REQUEST_URI}.%1 [L]
If /home/www/htdocs/news/article.html.2 exists,thenitwillbeservedasan
internalredirectandexitbeforethepublicationscriptisrun.Ifthefile
doesn’texist,thenprocessingwillfallthroughthelastrule,andwillbe
pickedupbythenormalpublicationdeliverymethod(presumablybyrun-
ningascriptspecifiedbyaScriptAliascalled /news).Thiswillworkforany
number of pages that end in an integer.204 | Information Servers
#98 Using mod_proxy on Apache for Speed
HACK
Likewise,theabsenceofaquerystringmightbemeaningful.Supposeyour
contentdeliveryscriptwillmanipulatecontentdependingonavarietyof
variables,butifnovariablesarespecified,itreturnspredictablecontend
(say,thedefaultviewofthefirstpageofanarticle.)Thisisanothercase
whereajudiciousRewriteRulecansaveyoutheunnecessarycomputing
overhead of dynamic rendering:
RewriteCond %{QUERY_STRING} =""
RewriteCond /home/www/static/%{REQUEST_URI} -f
RewriteRule .* /home/www/static/%{REQUEST_URI} [L]
Hereweassumethatifaquerystringexists,thenitmustnecessarilyinvolve
dynamiccontent.Otherwise,checkfirsttoseeifalocalcopyofthe
requestedcontentexists(under /home/www/static/)andifso,serveit.Ifthe
localcopydoesn’texist,thenprocessingfallsthroughtothenextsetof
directives(whichmostlikelyrunsthedynamiccontentgenerator).Aslong
asanexternalprocessmanagestheexpirationofoldcontentfromthecache
(via cron oradedicateddaemonthatwatchesforcontentchanges),thenthis
cachingwillbecompletelyseamlessfromtheuser’spointofview.Toexpire
oldcontent,simplydeletethecachedversion,andfuturehitswillbegener-
ated dynamically.
YoucandirectApachetotakeanyactionyoulikebasedonthequerystring.
Thepowerof mod_rewrite isreallylimitedonlybyyourimagination(and
the strength of your regex-fu).
See also:
•“Distributing Load with Apache RewriteMap” [Hack #99]
•“Ultrahosting:MassWebSiteHostingwithWildcards,Proxy,and
Rewrite” [Hack #100]
•Apache’smod_rewriteguidecanbefoundat http://httpd.apache.org/
docs/mod/mod_rewrite.html
HACK
#98
Using mod_proxy on Apache for Speed Hack #98
Offloadcomplexdynamicrequeststoanotherapache(oranothermachine
entirely)
AtremendousamountofefforthasgoneintooptimizingApachetomakeit
ripfilesfromthefilesystemandthrowthematincominghttprequestsas
quicklyaspossible.Unfortunately,sitesthatsolelyservecontentthatsitsin
fileswithinthefilesystemarenottypicallyverypopularsites.ThehugeUsing mod_proxy on Apache for Speed#98
Information Servers | 205
HACK
demandforinteractivecontenthasgivenbirthtomanyprojectsthatarespe-
cificallydesignedtogivetheenduserahighlycustomizable,dynamic
browsing experience.
Unfortunately,astheinteractivityofawebsiteincreases,performancetypi-
callygoesoutthewindow.Interactivecontentdemandsaprogramminglan-
guage,andinterpretingarbitrarycodeisveryexpensive(comparedto
servingstaticfiles).Oneclassic“worstcase”exampleisaPerlCGIthat
makesdatabaserequestsandsendsemailbyspawninganexternal sendmail.
Comparedtothetimeittakestoserveastaticfile,asingleCGIrequestof
thatsorttakesaneternity(andmanyrequeststhereforetakemanyeterni-
ties, bringing your entire server to a crawl).
Apachemodulessuchas mod_perl and mod_php (alongwithalargehelping
ofprogrammingsanity)cansignificantlyalleviatethepainofspawning
externalprocessesoneveryhit.ButevenwithinterpretersbuiltintoApache
itself,therateofservingdynamicrequestsrarelyapproachesthatofastatic
webserver.Onereasonforthisisthatmosthitssuffertheperformancepen-
altyofsupportingembeddedlanguages,regardlessofwhethertherequestis
dynamic or not.
Considerthecaseofalarge mod_perl installation.Afterhavingrunforafew
minutes,anindividualhttpdprocessmayconsume40or50MBofRAMas
Perlmodulesarecachedintomemory,forspeedyexecution.Nowimagine
thatarequestforasingle-pixeltransparentgifismade(asthisisonetrick
commonlyusedbywebdesignerstomakearbitrary-sizedblankspacesona
webpage,exacttothepixel).Intheabsoluteworstcase,supposethatall
availablehttpdprocessesarebusy,soApachedecidestospawnanother.The
servermustnowinvokethebehemothof mod_perl,creatingachildthat
takesup50MBofRAMandahugeamountofprocessingtime,justtoserve
asinglestaticfileofonlyafewbytes.Clearly,ifwecouldonlyinvokethe
shaggyshoggothof mod_perl whenabsolutelynecessary,wecouldreap
hugebenefitseverytimesomeonerequestedagraphic(orotherfilethatis
guaranteed to require no dynamic processing).
YoumightconsiderrunningtwoinstallationsofApache:onewithevery
module,bell,whistle,andvibra-slapthatyourdynamiccontentmanage-
mentsystemrequires,andanotherthatisasstripped-downandsimpleas
youcanpossiblymakeit.IfallrequestscomeintothelightweightApache,
thenwecanservestaticcontentasnormal,andproxydynamicrequeststo
the dynamic content servers.
Supposeyou’rerunningtheslimmed-downApacheonstandardport80,
andtheapplicationserveronport8088.Usingthefollowingrewriterule,206 | Information Servers
#99 Distributing Load with Apache RewriteMap
HACK
youcanproxyrequestsforanythingbutgraphicsandarchivestotheappli-
cationserver.PutthisatthebottomoftheVirtualHostentryforyoursite
(on the server running on port 80):
RewriteEngine On
RewriteCond %{REQUEST_URI} !.*\.(jpg|gif|pdf|png|zip|tgz|gz)$
RewriteRule ^/(.*) http://%{HTTP_HOST}:8088/$1 [P]
That [P] attheendoftheRewriteRulelinemeansproxy.Thiswillcausethe
lightweightApachetomakeaconnectiontothedynamicserverrunningon
8088andmaketherequestasiftheclientitselfhadmadeit.Whenit
receivesthereply,itpassesitalongtotheclientasifithadservedthe
requestitself.Asfarastheclientisconcerned,thereisonlyonewebserver:
thereallyfastonerunningonport80.Andaslongasthesitedefinitionsfor
thissiteinbothapacheinstallationspointtothesameDocumentRoot,this
method will work just fine.
Ifyourapplicationusescookies,youshouldalsoaddaProxyPassReverse
line just above the rewrite rules with something like this:
ProxyPassReverse / http://%{HTTP_HOST}:8088/
ThistellsApachetotranslatehttpheaderswhenproxying,sothatcookies
get propagated from the application server to the client as they should.
Ifyouhavethehardwareforit,youcouldhelpdistributetheloadbyusing
onemachinefortheproxyserverandanotherfortheapplicationserver.In
that case, try rules like this:
ProxyPassReverse / http://your.application.server.here/
RewriteEngine On
RewriteCond %{REQUEST_URI} !.*\.(jpg|gif|pdf|png|zip|tgz|gz)$
RewriteRule ^/(.*) http://your.application.server.here/$1 [P]
Ofcourse,withmultiplephysicalservers,you’llhavetokeepyourfilesys-
temsinsync(see“KeepingPartsofFilesystemsinsyncwithrsync” [Hack#41]),
orelsepeoplewillseeconflictingpageseverytimeyoursitegetsupdated.If
eventwoserversaren’tcapableofservingallofyourdynamiccontent,takea
lookat“DistributingLoadwithApacheRewriteMap” [Hack#99] forone
method of spreading load across as many application servers as you like.
HACK
#99
Distributing Load with Apache RewriteMap Hack #99
Scale to any number of web application servers with RewriteMap
Aswesawintheprevioushack,itispossibletotransparentlyservecontent
fromanyarbitrarywebserverusing mod_proxy and mod_rewrite.Rather
thanusinganexternalredirect [R],theproxytarget [P] makesthecontent
Download from Wow! eBook <www.wowebook.com>Distributing Load with Apache RewriteMap#99
Information Servers | 207
HACK
appeartocomefromtheserverfromwhichtheclientoriginallyrequested
content,withoutchangingtheURLlineintheclientbrowser.Aswesaw
earlier,oneapplicationofthistechniqueistospreadtheloadofserving
pagesacrosstwomachines,makingitpossibletoservemanymorehitsthan
a single machine can handle.
Butwhatifyouneedtoserveevenmoretraffic,tothepointthatasingle
applicationserverisnolongersufficient?Weneedamechanismthatwill
allowtheproxyserverstochoosefromanavailablepoolofapplicationserv-
ers,preferablyinawaythatwecandirectmorehitsatmorecapableserv-
ers. The RewriteMap directive gives us this functionality.
Here’s an example of how to specify a RewriteMap:
RewriteMap server rnd:/usr/local/apache/conf/servers.map
TheRewriteMaplinejustdefinesamapcalledserver,physicallyboundto
the servers.map fileinthe conf/ directory.Thernd:makesthisarandomized
plain text map.
The format of the servers.map file is very straightforward:
web www1|www2|www3|www4|www5
Theleft-handsideisanarbitraryvariablenamethatwecanusejustabout
anywhere in our configuration file, as in this RewriteRule:
RewriteRule ^/(.*) http://${server:web}.oreillynet.com/$1 [P]
Theright-handsideofthelinesofthe servers.map fileconsistsofstringsto
bereturnedinrandomorder,separatedbypipes.InthecaseoftheRewrit-
eRuleabove,arandomvalue(oneofwww1,www2,www3,www4,or
www5)willbechosenfromtheservers.mapandwillreplacethe ${server:
web} variable.Thisresultsinaproxyruleto http://www4.oreillynet.com/ (or
any of the other www* servers) with the original URI appended to it.
Theonlytroublewithatrulyrandommapisthatalloftheserversspecified
initwillreceiveapproximatelythesameamountoftrafficovertime.If
www1isaquad2.4GHzXeonwith8GBRAM,andwww2isa486SX/33
with4MBRAM,you’llprobablywanttodirectmoretrafficatwww1thanat
www2.Todothis,simplyspecifythemorecapableserversmultipletimesin
the servers.map:
web www1|www1|www1|www1|www2|www3|www3|www4|www5|www5
Inthisexample,www1willreceive4outof10hits,www2willreceive1out
of10,www3willreceive2,www4willget1,andwww5willpickupthe
remaining2.Youcanfinetunethisinrealtimebywatching top (oreven tl,as
discussedin“ConstantLoadAverageDisplayintheTitlebar” [Hack#59])on
everyapplicationserverwhilemanipulatingtheservers.map.Themtimeofthe
fileischeckedoneveryhit,andApachewillautomaticallyreloadthemap208 | Information Servers
#100 Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite
HACK
wheneveritchanges,withoutneedingarestart.Thismeansthatanintelligent
systemsmonitoringtoolcanwatchallofwww*androtatethemoutofthe
mapinrealtimewheneveritfindsawebserverthat’shavingtrouble.Ifcare-
fullyplanned,thiscanmeanwebserviceswithzerodowntime,routingaround
problem servers, all without ever having to restart Apache.
Workingwithmultipleapplicationserverscanbetrickybusiness.Obvi-
ously,you’llneedtokeepyourfilesystemsinsync,usingNFSorpossibly
rsync,asdiscussedin“ConstantLoadAverageDisplayintheTitlebar” [Hack
#59].Buteventhebestplannedserverschemescansometimesgoawry.What
happensifyounoticeanInternalServerErrorthatgoesawayifyouclick
Reload?Oneoftheapplicationservershasdevelopedaproblem,andthere’s
nowaytotellwhichoneishavingtrouble(withoutconsultingtheerrorlog
oneverymachine).Here’sahandyrewriterulethatwillletyouselectwhich
application server to steer a particular request:
RewriteCond %{QUERY_STRING} appserver=(.*)
RewriteRule ^/(.*) http://%1.oreillynet.com/$1 [P,L]
Nowyoucanselectaparticularapplicationserverbyaddingaquerystring
to the URL in your browser, like this:
http://www.oreillynet.com/some/path/?appserver=www3
Thiswillhelpyouquicklytrackdowntheserverwiththeproblem.Com-
binedwithfront-endproxyservers,roundrobinDNS,andreplicateddata-
bases,thismakesaverystraightforwardconfigurationthatwillscaletovery
large installations.
See Also:
•Apachemod_rewritedocumentationat http://httpd.apache.org/docs/mod/
mod_rewrite.html
•Proxy Server Hack
•“Using rsync over ssh” [Hack #38]
•“Distributing Server Load with Round-Robin DNS” [Hack #79]
•“Setting Up Replication in MySQL” [Hack #82]
HACK
100
Ultrahosting: Mass Web Site Hosting with
Wildcards, Proxy, and Rewrite Hack #100
Support thousands of internal web servers without lifting a finger
SupposeyouhavealargeprivatenetworkhidingfromtheInternetbehinda
NAT router. Your network layout looks something likeFigure8-3.Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite#100
Information Servers | 209
HACK
Youwanttobeabletoallowanyoneonyourprivatenetworktosetuptheir
ownwebserver.Butlikeallgoodnetworkadministrators,youaresmartand
lazyanddon’twanttofiddlewithupdatingforwardingrulesonyourfire-
walleverytimesomeoneneedstomakeachange.Throughthecarefuluseof
namedvirtualhosts, mod_proxy,and mod_rewrite,youcanreducethe
administrativeoverheadofyourentirenetworktosimpleDNSupdates.
Thenthereislittlekeepingyoufromdelegating that responsibilitytothe
departments that wanted the web servers in the first place.
Tostart,you’llneedApacherunningonyourgatewaymachine,with mod_
rewrite and mod_proxy installed.You’llalsoneedaDNSserverrunning
yourowntop-levelinternaldomain(asdiscussedin“RunningYourOwn
Top-LevelDomain” [Hack#80]).We’llassumethatyouowntheInternet
domain shelbyville.com,andhavetheinternalTLDof .springfield alreadyset
up, serving your internal machines.
Add the following to the Apache configuration on your gateway machine:
Port 80
BindAddress *
NameVirtualHost *
<VirtualHost *>
ServerName mux.shelbyville.com
ServerAlias *.shelbyville.com
RewriteEngine On
RewriteCond %{HTTP_HOST} (.*).shelbyville.com
RewriteRule ^/(.*) http://%1.springfield/$1 [P]
</VirtualHost>
Figure8-3.Typical corporate lans use private addressing internally and at least one
Internet gateway providing Network Address Translation
IS Department
(wiggum.springfield)
10.0.5.0/24
Corporate
(krabappel.springfield)
10.0.6.0/24
Sales
(syzlak.springfield)
10.0.7.0/24
Research
(frink.springfield)
10.0.8.0/24
Facilities
(willy.springfield)
10.0.9.0/24
Linux
gateway
gateway.springfield
10.0.0.1 Internet
208.201.239.36
gateway.shelbyville.com210 | Information Servers
#100 Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite
HACK
Briefly, this configuration translates to:
•Listen on any available IP address, on port 80
•Accept http requests for any host ending in .shelbyville.com
•Strip shelbyville.com from the http host that was requested
•Makeaconnection(viaProxy)towhat’sleftofthehostnamewith .spring-
field appended to it
•Post the original URI and return the results to the client
Thisworksbecausehostsarespecifiedbynameinthehttpheaderunderthe
http1.1specificationandaren’ttiedtoaparticularIPaddress.Apachewill
lookupthecontentsof%1(everythingbefore .shelbyville.com)viathesys-
temresolverandattempttoconnecttoit.SincethegatewayisusingaDNS
serverthatservesyour .springfield TLD,itwillproxytotheproperinternal
host.
Forexample,supposethattheoriginalrequestwasfor http://jimbo.shelbyville.
com/index.html.AftertheRewriteCondline,the%1variablesimplycontains
jimbo.Itthenattemptstoproxyto%1(akajimbo)with .springfield appended
toit.Theresult?Aproxyrequesttotheinternalwebserver jimbo.springfield,
with the original URI passed along as if the gateway weren’t even there.
Thisisthesimplestconfiguration,butitwillbreakinternalserversthat
requirecookies.Tosupportcookiesonserversresidingontheinternalnet-
work, try something like this:
<VirtualHost *>
ServerName mux.shelbyville.com
ServerAlias *.shelbyville.com
RewriteEngine On
RewriteCond %{HTTP_HOST} (.*).shelbyville.com
RewriteRule (.*) $1 [E=WHERETO:%1.springfield]
ProxyPassReverse / http://%{ENV:WHERETO}/
RewriteRule ^/(.*) http://%{ENV:WHERETO}/$1 [P]
</VirtualHost>
Thisusesa“fake-out”RewriteRulethatisonlycalledtoinvokethesideeffect
ofsettingthe %{WHERETO} environmentvariable.Thisgetssettotheoriginal
requestedhttphostwith .shelbyville.com strippedoff,butwith .springfield
appended.Weneedtodothattobeabletofeedtheamendedhostnameto
ProxyPassReverse.Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite#100
Information Servers | 211
HACK
BymanipulatingtheDNSconfigurationfor .shelbyville.com and .springfield,
youcanbringinternalwebserversupanddownatwill,withoutevertouch-
ingtheApacheconfigurationonthegateway.Ofcourse,tomakethejob
even easier, you could use wildcard DNS for .shelbyville.com:
*.shelbyville.com IN A 12.34.56.78
(Naturally,substitutingtheexternalIPaddressofthegatewayfor12.34.56.
78).Now,requestsfor anything.shelbyville.com willbefedtothegateway
andproxied,withouteverchangingthezonefileagain.Thatjustleaves
internalDNSmaintenance(for .springfield).Thesimplestwaytodispense
withthatresponsibilityistodivideitintomultiplesubdomainsanddeferto
otherinternalDNSservers.Forexample,youcouldputsomethinglikethis
in your named.conf:
zone "wiggum.springfield" {
type slave;
file "wiggum.db";
masters { 10.42.5.6; };
};
zone "krabappel.springfield" {
type slave;
file "krabappel.db";
masters { 10.42.6.43; };
};
zone "syzlak.springfield" {
type slave;
file "syzlak.db";
masters { 10.42.7.2; };
};
andsoon.NoweachdepartmentcanhavetheirownmasterDNSserverand
canaddnewhosts(andthereforenew,Internet-readywebservers)without
evendroppingyouanemail.Theycangettotheirnewserversfromthe
Internetbybrowsingtoanaddresslike http://ralph.wiggum.shelbyville.com/,
whichtranslatesto http://ralph.wiggum.springfield/ internallyandisulti-
matelylookedupfromtheDNSserverforthedivisionthatisresponsiblefor
thewiggumsubdomain.AsfarastheInternetusersareconcerned,theinfor-
mationcamedirectlyfromthegatewaymachine,andaren’tevenawarethat
.springfield even exists.
Now, just what will you do with all of the time that this saves you?213
We’d like to hear your suggestions for improving our indexes. Send email to index@oreilly.com.
Index
` (backtick),14, 20
^D,21
. (dot) and CVS tags,56
/ (forward slash),66
| (pipes), efficiency of,15
? (question mark) and top,119
; (semicolon) and Perl one-liners,151
~ expansion facility,21
Numbers
1024th cylinder limit, kernel install,73
A
access
ACL (access control list), local
host,161
hosts, assigning trust-levels to,89
lost root password,6
ad referral tracking,188–191
advanced routing tools,98
anon_auth_module,191
Apache,157, 182–211
ad referral tracking,188–191
anon_auth module, enabling,191
Apache Toolbox script,182–185
installation,183–185
supported applications,182
apachectl,188
certificate generation and signing
requests,194–196
certificates, distribution to client
browsers,199–200
configuration files, automatic
updates across a network,28
ftp servers, mimicking,191–193
controlling access,191
httpd.conf,187
configuring display of full
filenames,185
IfDefine, configuration changes
using,186–188
IfModule directive,186, 187
mass web site hosting,208–211
mod_proxy, speeding content
delivery using,204–206
mod_rewrite,201–204
multiple installations, increasing
efficiency using,205
RewriteMap, load distribution
using,206–208
server logs, rotation and
compression,193
serving cookies,206
sites with a shared
DocumentRoot,201–203
ssl keys and certificates, using,198
URL query strings, content delivery
using,203
archiving with pax,67–72
restoring,68
and renaming,69
at, disabling,23
automount,4214|Index
B
backtick (`),14, 20
backups,64–87
of the boot sector,72
CDR/CDRWs,84–87
over a network,65–67
rsync,66
scp command,65
tar, using over ssh,65
with pax,67–72
incremental backups,71
restoring compressed archives,68
snapshot-style,79–84
balance-push.sh script,77
bash
~ expansion facility,21
clearing history,22
files, breaking into chunks,153
login shells, csutomizing,22
manpage,20
shell environment,
customizing,20–22
tab completion,15
.bash_logout,22
.bashrc,22
network configuration,66
binary files
chunking,153–155
reassembling,154
BIND,157, 158–169
caching DNS with local domain
authority, setup,165
running in a chroot jail (in V.
9),158–160
server load distribution (in V. 9),167
views (in V. 9),160–165
match-clients substatement,161
named.conf,161
named.conf, primary master,164
named.conf, slave name
servers,163
syntax,160
zones, defining,162
BIOS, 1024th cylinder boot sector
limit,73
blacklists,89–91
boot messages, viewing,40
boot parameters,7–9
boot sector, backing up,72
boot time configuration,2, 3–10
1024th cylinder limit,73
removing superfluous services,3
unnecessary drivers,
eliminating,40–42
C
cacert.pem,197
cakey.pem,197
CA.pl utility,197
CDPATH variable,21
CDR/CDRWs,84
CD-ROMs,85–87
Certificate Authorities,196
certificates, acquiring from,194
creating your own,196–198
keys, file locations,197
public keys, distributing to
clients,199–200
chargen,5
chattr,18
chroot jail,158–160
ci command (RCS),48
cmdline files, viewing,33
co command (RCS),48
CodeRed virus, filtering,93
command line,2, 10–46
complex commands, creating,12–15
watch, automating a repeated
command using,114
compiles,parallelbuilds(multiprocessor
systems),19
comsat,5
Concurrent Versioning System (see CVS)
.config file,41
configuration files, updating with
make,27
console= (boot parameter),8
Control D,21
cp command
-a switch,79
-l switch,79
cpio
pax and,67
crontab
automating snapshot-type
backups,80
setuid and,23Index|215
CVS (Concurrent Versioning
System),52–63
anonymous repositories, creating,62
branching development,59
changing CVS servers,153
commands,55–59
config trees,52
cvs-makerepos (Debian),53
default editor, setting,57
directories
removing,59
structure,55
documenting changes,57
environment variables for,54
files
merging,58
permissions, setting,61
watching and locking,60
modules
checking out,54
importing new,53
pserver access method,62
release tags,53
repository, creating,53
security,60
connecting to remote
machines,54
developer machines,61
SSH setup,61
uses,52
vendor tags,53
$CVSEDITOR environment variable,57
$CVSROOT environment variable,54,
55
CVSROOT/readers file,63
$CVS_RSH environment variable,
setting to ssh,54
$CVSUMASK environment variable,61
D
daemons
init, creating with,9–10
databases (see MySQL)
data.tar,173, 174
dd,154
denial-of-service attacks, preventing
with iptables,92
der certificate format,199
$DEST_IP,92
directories
backup of home,67
copying recursively,70
CVS,55
removal from,59
identifying largest users of disk
space,31
replication,65
disk age analysis,125–127
disk space, determining largest users,30
diskage script,125
dmesg command,40
DNS (domain name service)
caching, with local domain
authority,165
logging on syslog,159
round-robin load distribution,167
DocumentRoot, sharing among several
sites,201–203
domain name queries, automated
pattern matching,29
downloads from Apache servers,
enabling,191–193
drivers
built-in vs loadable modules,40
unnecessary, eliminating,40–42
drives, (E)IDE, optimizing
performance,43–46
E
echo,5
email, securing with ssh,146
environ files, viewing,33
environment variables
bash, customizing in,20–22
for CVS,54
viewing with /proc,31–34
errors
diagnosing (example),12–15
ethereal and network debugging,100
ext2 filesystem
immutable files, creating in,17
ext3 filesystem
immutable files, creating in,17
$EXT_IFACE,92
extract-table script,175
Download from Wow! eBook <www.wowebook.com>216|Index
F
fifo, logging of debug info to,112
file command,68
file descriptor,11
file descriptors
3-9,11
closing,12
filenames
special characters in, handling,15
files
associated processes,
finding,116–119
breaking into arbitrary
chunks,153–155
reassembling,154
immutability flag, setting,17
immutable, creating in ext2/ext3
filesystems,17
rsync and synchronization across a
network,74–79
security, enhancing,18
filesystem
monitoring changes to,125–127
filesystems
ISOs (CD-ROM data format),85–87
find used with xargs,17
finger,5
finger, security,24
firewalls,88–91
access, setting trust-levels,89
forged packets, blocking,91
ftp,5
mimicking with Apache,191–193
G
Generic Routing Encapsulation
(GRE),99
go-ogle script,123
gpm, security,24
GRE (Generic Routing
Encapsulation),99
grep, colorizing output,155
grep -v ^# /etc/inetd.conf,4
H
hacks,xv
hdparm,43–46
online resources,46
hdX= (boot parameter),8
High Availability Linux project, send_
arp utility,129
history command
clearing your history (bash),22
home directory, backing up,67
hosts, access by, setting,89
httpd.conf
display of full filename in indexes,
configuration,185
httpd.conf, diagnosing problems with
RCS,49
httptop,132–138
required Perl modules,133
I
-i, immutable flag, chattr and lsattr,18
IDE hard drives, performance
optimization,43
identd,5
IfDefine,186–188
IfModule,186, 187
inbound port scans, blocking,93
incremental backups using pax,71
inetd,4
inetd.conf,4
CVS and,63
information servers,157–211
(see also Apache)
(see also BIND)
(see also MySQL)
Init
creating persistent daemons
using,9–10
init,9
inittab,9
Intel Physical Address Extension (PAE)
mode,42
$INT_IFACE,92
I/O support parameter, hdparm,44
IP aliasing,127–129
IP masquerading,91
IP tunneling,97
IPIP,99
IPIP tunneling
GRE, compared to,99
iplimit patch,93
iptables
custom chains, using in,96
ipchains, compared to,92Index|217
kernel patches, features enabled
with,93
port forwarding and,94
syslog port, protecting,114
TCP flags and,93
TCP ports, forwarding,94
transparent Squid proxy,92
iptables command,88–94, 96–101
firewalls, creating,88–94
IP masquerading,91
ISOs (CD-ROM data format),85–87
K
kernel
.config file,41
drivers,40–42
modules, build considerations,41
modules, built-in vs. loadable,40
minimum boot functionality,41
RAM support, configuring,42
viewing running processes,31–34
L
LILO, 1024th cylinder boot sector
limit,73
Linux
operating systems,1
optimization,2
Linux servers,1
(see also servers)
load average, script for displaying in
titlebar,120
log files,111
logflume.pl script,193
logging
DNS on syslog,159
login shells,22
logins
ssh, using to facilitate,139–142
turbo-mode,141
lp, security,24
lpd,5
lsattr,18
lsof,116–119
-a switch,118
-c switch,117
devices, specifying,117
-i switch,118
-p switch,117
M
mail spool, determining largest user,30
make command, for file updates,27
master,172
mem= (boot parameter),8
mince script,154
mod_proxy,204–206
mod_rewrite,201–204
query strings, using for content
delivery,203
RewriteLog,203
mod_sql module,178
monitoring,111–138
httptop,132–138
IP monitoring and
takeover,127–129
MAC address updating,128
load averages, displaying,120
MySQL threads,169
ntop,129–131
scanning the networks
machines,123–125
syslog and,111–114
top,119
watch,114
web traffic,132–138
mouse drivers, security of,24
movein.sh,149
mtop,169
available commands,170
e (explain),171
k (kill),171
user, creating for,171
multcount support parameter,
hdparm,44
multicast packet tunneling,99
multiprocessor systems, improving
compile efficiency,19
MySQL,157, 169–182
dedicated servers,
optimizing,180–182
glibc,181
kernel changes,181
maximum filehandles,
expanding,181
MySQL build,181
risks,180
mod-sql module,178
monitoring,169218|Index
MySQL (continued)
proftpd and user
authentication,178–180
replication,172
typical problems,174
write requests,172
replication user,172
server tuning,175
indices,176
my.cnf,177
mysqlcheck -o database,176
mysqld, renicing,176
safe_mysql script,176
server variables, checking,177
servers, master and slaves,172
table restoration from a dump,175
users, creating,179
N
named,4, 158–160
changing ownership from root,158
named.conf for views,161
named.conf
for caching DNS with local domain
authority,165
NAT (network address translation),
circumvention with
vtun,101–106
netfilter patches,93
netstat,115
netstat -lp,4
networking,88–110
firewalls,88–91
networks
admin tasks, automating,208–211
and Linux servers,1
scanning of machines,123
NFS services,4
NFSfilesystem, file-syncing with,74–79
ngrep,121–123
syntax,122
Nimda virus, filtering,93
n>&m operator,11
n>&m operator (Bourne shell),11
nmap,123–125
-O switch,124
nmbd,4
nosmp (boot parameter),8
ntop,129–131
admin password, setting,130
databases, initializing,130
SSL, running on,130
O
OpenSSH,139
X11 forwarding, enabling,145
OpenSSL, key generation,195
OS fingerprinting code,123
P
passwords
lost root, getting around,6
ssh keys, compared to,140
patch-o-matic patches,93
pax (portable archive exchange),67–72
archives, restoring,68
and renaming,69
-c (exception) switch,72
cpio and,67
directories, recursive copying,70
home directory, backup using,67
incremental backups,71
-n switch,72
-r switch,68
risks of misuse,70
skipping files on restore,72
tar and,67
-z switch,68
pem certificate format,199
Perl
-e switch,151
global search and replace,151
-i switch,152
-p switch,152
pgrep,35
ping
setuid and,25
pkill,35
portmap,4
ports
checking for Listen state,115
restricting access,89–91
ssh, forwarding over,146–148
syslog port, protecting,114
preferences, automating with
scripts,149Index|219
privileges, minimizing security risks
of,25
/proc,31–34
evaluating system security,34
processes for open sockets and files,
identifying,116–119
procps,34–36
.profile
ssh-agent, inclusion in,144
proftpd and mysql
authentication,178–180
prompts
colorcoding in bash shell,20
setting default,20
ps
security and,34
ps ax,4
PS1 variable,20
psd patch,93
pserver,62
installing,62
remote use,63
Q
QUERY_STRING, using for ad
tracking,188–191
R
r* commands, setuid and,24
RAM,42
64 GB, support to,42
960MB, support above,42
RCS (revision control system),48–52
checking out files,48
-r switch,50
diagnosing problems with,49
initializing a file,48
locking files,48
logging changes,49, 50–52
log entries, correspondence to
versions,51
rcs2log,50–52
-v switch,51
rcsdiff,49, 51
repository, creating,48
unlocking files,48
real-time netork stats,
monitoring,129–131
referral-report.pl script,190
release tags (CVS),53
replication in MySQL, purpose,172
restricted port policies,89–91
revision control,47–63
(see also CVS)
(see also RCS)
revision control system (see RCS)
RewriteMap,206–208
servers.map file,207
rexec,5
rgc (Regex Colored Glasses) script,155
rinetd,94
rlogin,5
ro (boot parameter),8
root= (boot parameter),7
root password
loss of, repairing,6
rootservers.cache,166
rpc.mountd,4
rpc.nfsd,4
rp_filter,91
rsync,66
--delete flag,67
“downward spiral” condition,75
exclude-from feature,75
excludes lists,82
file syncing with,74–79
Perl script in cron,76–79
snapshot-type backups and,79
ssh, using over,66
rw (boot parameter),8
S
safe_mysqld script,176
Samba daemons,4
scp command,65
scripting,149–156
automatic configuration of
preferences on remote
systems,149
files, breaking into chunks,153–155
reassembling,154
global search and replace using
Perl,151
logfiles, colorizing data in,155
security
certificates, trustworthiness,199
closing access to ex-users,38–40
CVS and remote connections,54220|Index
security (continued)
enhancing with non-standard
TLDs,168
evaluating using /proc,34
files, enhancement of,18
history, clearing (bash),22
name resolution and,158
setuid/setgid binaries,
eliminating,23–25
semicolon (;) and Perl one-liners,151
send_arp,129
sendmail
configuration files, automatic
updating,27
servers
distributing load with round-robin
DNS,167
monitoring (see monitoring)
updating asynchronously,74
servers.map file,207
services, removing unnecessary,3
setuid and setgid binaries,
finding,23–25
setuid and sudo,27
single (boot parameter),7
skill,35
slaves,172
smbd,4
snapshot backups,79–84
snapshot-type backups
rotating snapshots,80
Snark rule,21
snice,35
sockets
identifying listeners,4
sockets and associated processes,
identifying,116–119
SQLAuthTypes,179
SQLConnectInfo,179
Squid proxy,92
src directory, CVS,55
SSH
CVS, using for,61
ssh,65–67, 139–148
backups with tar,65
CVS and remote logins,54
key generation,140
keys, compared to passwords,140
keys, usefulness of,66
logins, using for,139–142
turbo mode,141
port 22,90
ports, forwarding,146–148
private keys, protecting,150
rsync over,66
security of,24
ssh-agent,142–144
GUI, running in,144
ssh-to script,141
traffic, securing with,146
vtun, using with,148
X11 traffic, forwarding over,145
SSL and ntop,130
standard error
redirecting,10
standard output
redirecting,10
sudo,25–27
setuid and,27
syntax,26–27
groups,27
users,26
syslog,111–114
DNS logs and,159
filtering facilities and priorities,111
MARK messages,113
protecting the syslog port,114
remote logging, risks of,113
syslog.conf,112
system administration
automated logout of idle users,22
delegation of responsibilities,25
environment, replicating across a
network,149
make, system updates using,27
network monitoring (see monitoring)
processes, manipulating with
procps,34
runaway processes,
preventing,36–38
ssh, using for logins across the
network,139
user accounts, closing,38–40
system administrators,xvi, 1
system resource limits, setting,36–38Index|221
T
tab completion,15
takeover script,128
tar,66
backups over ssh,65
directories, recursive copying,70
pax and,67
tcpdump and network debugging,100
telnet,5
terminal windows
titlebar, customizing,20
tl script,120
TLDs (top level domains),168
TLDs (top-level domains)
running your own,168
TLDs (top-level domains), pattern
matching to,29
TMOUT variable,22
top,119
? (question mark),119
filtering of results,119
sort criteria, specifying,119
top command,34
traceroute
setuid and,25
transparent Squid proxy,92
tun,101
tunneling
IP tunneling,97
NAT, circumvention with vtun over
ssh,101–106
U
unmaskirq parameter, hdparm,45
URL query strings and content
delivery,203
users
closing user accounts,38–40
using_dma parameter, hdparm,45
UUCP and setuid/setgid binaries,25
V
vendor tags (CVS),53
views (see under BIND)
VirtualHosts, monitoring,133
virus filters,93
visudo,26
vmstat,35
vtun,101–110
NAT, circumvention ssh,101–106
ssh, using over,105
vtund server, launching,103
vtund.conf,102–110
client-side,103
automatic generation via Perl
script,106–110
server-side,102
W
wall
setgid and,24
watch,114
web servers
diagnosing error conditions
(example),12–15
web sites, performance costs of
interactivity,205
web traffic, monitoring,132–138
whitelists,89–91
whitespace in filenames, handling,16
write
setgid and,24
X
X11 traveling, forwarding over ssh,145
xargs,16
find, used with,17
Download from Wow! eBook <www.wowebook.com>Colophon
Ourlookistheresultofreadercomments,ourownexperimentation,and
feedbackfromdistributionchannels.Distinctivecoverscomplementour
distinctiveapproachtotechnicaltopics,breathingpersonalityandlifeinto
potentially dry subjects.
Thetoolonthecoverof LinuxServerHacks isanax.Anaxisachopping
toolbasedononeofthesixsimplemachinesofphysics:thewedge.Though
theaxisoneoftheearliestman-madetools,datingbackanywherefrom
100,000to500,000,itssimplicityandefficiencymakeitindispensableto
this day.
SarahShermanwastheproductioneditorandcopyeditorfor LinuxServer
Hacks.ColleenGorman,MaryBrady,andClaireCloutierprovidedquality
control.JohnBickelhauptwrotetheindex.LinleyDolbyprovidedproduc-
tion assistance.
EdieFreedmandesignedthecoverofthisbook.Thecoverimageisanorig-
inalphotographfromtheCMCDcollection.EmmaColbyproducedthe
coverlayoutwithQuarkXPress4.1usingAdobe’sITCGaramondand
Helvetica Neue fonts.
DavidFutatodesignedtheinteriorlayout.Thisbookwasconvertedto
FrameMaker5.5.6withaformatconversiontoolcreatedbyErikRay,Jason
McIntosh,NeilWalls,andMikeSierrathatusesPerlandXMLtechnolo-
gies.ThetextfontisLinotypeBirka;theheadingfontisAdobeHelvetica
NeueCondensed;andthecodefontisLucasFont’sTheSansMono
Condensed.Theillustrationsthatappearinthebookwereproducedby
RobertRomanoandJessamynReadusingMacromediaFreeHand9and
Adobe Photoshop 6. This colophon was written by Linley Dolby.

Tips&Tricks

LINUX SERVER HACKS

No comments:

Post a Comment